*! version 1.15.8.19 19ago2015
help parallelalso see: miparallel

Title

parallel -- Stata module for Parallel computing

Index

Sections

1. Syntax Command syntax.

2. Description Command description.

3. Details How does parallel works.

4. Parallel Append Using -parallel append- syntax.

5. Caveates Things to consider before using parallel.

6. Technical note Some details under the hood.

7. Examples Some examples using parallel

8. Saved results A list of parallel's save results

9. Development Up-to-date version and bug reporting

10. Source code parallel's (MATA) source code

Available commands

1. parallel setclusters Setting the number of clusters.

2. parallel do Parallelizing a do-file.

3. parallel : (prefix) Parallelizing a Stata command (parallel prefix).

4. parallel bs Parallel bootstrapping.

5. parallel sim Parallel simulate.

6. parallel append Multiple file processing and appending.

7. parallel clean Removing auxiliary files.

8. parallel seelog Checking out clusters' logfiles.

9. parallel version Query parallel current version.

10. parallel break Query if the mother process has requested to break.

1. Syntax

Setting the number of clusters (threads/processors)

parallel setclusters # [, force
statapath(stata_path)
gateway(gateway_path)
includefile(filename)]

Parallelizing a do-file

parallel do
filename [, by(varlist) force keep
keeplast programs(namelist) mata
noglobal seeds(numlist) nodata
randtype(current|datetime|random.org)
timeout(#)
processors(#)
setparallelid(pll_id)]

Parallelizing a Stata command (parallel prefix)

parallel [, by(varlist) force keep
keeplast programs(namelist) mata
noglobal seeds(numlist) nodata
randtype(current|datetime|random.org)
timeout(#)
processors(#)
setparallelid(pll_id)]:
command

Parallel bootstrapping

parallel bs
[, expression(exp_list) programs(namelist)
mata noglobal seeds(numlist)
randtype(current|datetime|random.org)
timeout(integer)
processors(integer)
bs_options
] [: command]

Parallel simulate

parallel sim
, reps(#) [
expression(exp_list) programs(namelist)
mata noglobal seeds(numlist)
randtype(current|datetime|random.org)
timeout(#)
processors(#)
sim_options
] [: command]

Multiple file processing and appending

parallel append
[file(s)] ,
do(cmd|dofile) [in(in) if(if)
expression(expand expression (see details))
processors(#) programs(namelist)
randtype(current|datetime|random.org)
timeout(#)]

Removing auxiliary files

parallel clean [, event(pll_id) all force]

Checking out clusters' logfiles

parallel seelog [#] [, event(pll_id)]

parallel printlog [#] [, event(pll_id)]

Query parallel current version

parallel version

Query if the mother process has requested to break

parallel break

Setting the number of clusters
force Overrides the 8 clusters restriction (see the
WARNING in description).

statapath File path. parallel tries to automatically identify
Stata's exe path. By using this option you will override this and force
parallel to use a specific path to stata.exe.

gateway File path. For Windows, a file that a Cygwin process
is listening to in order to execute the parallel instances. See example below.

includefile File path. This file will be included before parallel commands
are executed. The target purpose for this is to allow one to copy over preferences that
parallel does not copy automatically.

Byable parallelization
by Varlist. Tells the command through which observations the current dataset
can be divided, avoiding stories (panel) splitting over two or more clusters.

force When using by, parallel checks whether if the dataset
is properly sorted. By using force the command skips this check.

Keeping auxiliary files (use with caution)
keep Keeps auxiliary files generated by parallel.

keeplast Keeps auxiliary files and remove those last time saved
during the current session.

Passing Stata/Mata objects
programs A list of programs to be passed to each cluster.

mata If the algorithm needs to use mata objects, this option allows
to pass to each cluster every mata object loaded in the current session (including functions).
Note that when mata objects are loaded into the child processes they will have different
locations and therefore pointers may no longer be accurate.

noglobal Avoid passing current session's globals to the clusters.

Simulation options
seeds Numlist. With this option the user can pass an specific seed to be
used within each cluster.

nodata Tells parallel not to use loaded data and thus
not to try splitting or appending anything.

randtype String. Tells parallel whether to use the current seed
(-current-), the current datetime (-datetime-) or random.org API (-random.org-) to
generate the seeds for each clusters (please read the Description section).

Parallel bootstrap
expression An exp_list to be passed to the bootstrap command.

bs_options Further options to be passed to the bootstrap command.

Parallel simulate
expression An exp_list to be passed to the simulate command.

sim_options Further options to be passed to the simulate command.

Multiple file processing and appending
do Stata cmd or dofile.

expression String. Expression representing file names in the form of
"%fmts, numlist1 [, numlist2 [, ...]]"

Removing auxiliary files
event Integer. Specifies which executed (and stored) event's files should be removed.

all Tells parallel to remove every remnant auxiliary files generated by it
in the current directory.

force Forces the command to remove (apparently) in-use auxiliary files. Otherwise
these will not get deleted.

Other options
timeout Integer. If a cluster hasn't start, how much time in seconds
does parallel has to wait until assume that there was a connection error and thus
the child process (cluster) won't start. Default value is 60.

processors Integer. If running on StataMP, sets the number of processors
each cluster should use. Default value is 0 (do nothing).

setparallelid Programmers' option. Forces parallel to use an specific
id (pll_id) (see Technical Notes).

2. Description

-parallel- allows to implement parallel computing, without having StataMP,
substantially reducing computing time. Specially suitable for bootstrapping and
simulations, parallel includes out-of-the-box tools for implementing such
algorithms.

In order to use -parallel- it is necessary to set the number of desired clusters
with which the user wants to work with. To do this the user should use -parallel setclusters-
syntaxes, replacing # with the desired number of clusters. Setting more clusters
than physical cores the user's computer has it is not recommended (see the WARNING
in description).

-parallel do- is the equivalent (wrapper) to -do-. When using this syntax
parallel runs the dofile in as many clusters as there where specified by the user, this is,
start $PLL_CLUSTERS Stata instances in batch mode. By default the
loaded dataset will be split into the number of clusters specified by -parallel setclusters-
and the do-file will be executed independently over
each and every one of the data clusters, so once after all the parallel-instances
stops, the datasets will be appended. In order to avoid loading the current
dataset in the clusters, the user should specify the -nodata- option.

-parallel :- (as a prefix) allows to, after splitting the loaded dataset,
execute a stata_cmd over the specified number of data clusters in order to
speed up computations. Like -parallel do-, after all the parallel-instances
stops, the datasets will be appended.

-parallel bs- and -parallel sim- are parallel wrappers for the commands
-bootstrap- and -simulate-. Specially suited for these algorithms,
-parallel- allows conducting embarrassingly parallel computing. In terms of
syntax, besides cmd names, the only difference that these two commands have with
their serial versions is how are expressions passed (please refer
to the examples section for this).

Every time that -parallel- runs several auxiliary files are generated which,
after finishing, are automatically deleted. In the case that the user sets -keep-
or -keeplast- the auxiliary files are kept, thus the syntax -parallel clean-
becomes handy. With -parallel clean- the user can remove the last generated
auxiliary files (default option), an specific parallel instance files (using
#pll_id number), or every stored auxiliary file (with -all-). For
security reasons, in-use auxiliary files will not be deleted unless the user
specifies it through the option force, which will override not deleting
in-use auxiliary files (see the Technical note section for
more information about this).

In the case of handling multiple files (because it is, for example,
a big dataset divided into multiple dta files), -parallel append- becomes handy
as it allows the user to process them simultaneously. By providing a list of files
and a Stata cmd or dofile, -parallel append- opens and executes the cmd/dofile
within each file, stores each file results and appends them into a single file.
Also, if the files to be processed have a pattern base name, the user can provide
-parallel append- with an expression representing the list of files to be
processed; for information on how to use this feature, see the section
Parallel Append.

If the user needs to allow the mother instance to break, users can insert
the command -parallel break- (stata) and the functions
-parallel_break()- or -_parallel_break()- (mata) in some parts of his
code. These commands/functions check whether the user has pressed the
-break- key in the mother process.

Given N clusters, within each cluster -parallel- creates the macros
pll_id (equal for all the clusters) and pll_instance (ranging
1 up to N, equalling 1 inside the first cluster and N inside the last cluster),
both as globals and locals macros. This allows the user setting different
tasks/actions depending on the cluster. Also the global macro PLL_CLUSTERS
(equal to N) is available within each cluster. For an example using this
macros, please refer to the Examples section.

As by now, -parallel- by default automatically identifies Stata's
executable file path. This is necessary as it is used to run Stata in batch mode
(the mainstream of the module). Either way, after some reports, that file path is not
always correctly identified; where the option -statadir- in -parallel setclusters-
can be used to manually set it.

In the case of pseudo-random-numbers, the module allows to pass different seed for
each cluster (child process). Moreover, if the user does not provide a numlist of
seeds, -parallel- generates its own numlist of seeds using three different options:
(1) based on the current seed; (2) using the current datetime and user as a seed to
generate each seed, restoring the original seed afterwards; or
(3) using random.org API (requires internet connection) to directly generate each
seed (also restoring the original seed afterwards). -parallel- saves a macro
with the used seeds in the r(pll_seeds) macro.

WARNINGS (a) For each cluster -parallel- starts a new Stata instance (thus
running as many processes as clusters), this way, should the user set more clusters
than cores the computer has, it is possible that the computer freezes.
(b) Unless the user includes -parallel break- within the task (do-file or program),
by this time -parallel- can not stop running the clusters by itself, what
implies that, in the case of any of the clusters starts a endless loop, stoping the
clusters should be done manually by the user by terminating them from the OS's tasks manager.

3. Details

Inspired by the R library ``snow'' and to be used in multicore CPUs
, -parallel- implements parallel computing methods through an OS's shell
scripting (using Stata in batch mode) to speedup computations by splitting the
dataset into a determined number of clusters in such a way to implement a
data parallelism algorithm.

The number of efficient computing clusters depends upon the number of physical
cores (CPUs) with which your computer is built, e.g. if you have a quad-core
computer, the correct cluster setting should be four. In the case of simultaneous
multithreading, such as that from
Intel's hyper-threading technology (HTT),
setting -parallel- following the number of processors threads, as it was expected,
hardly results into a perfect speedup scaling. In spite of it, after several tests
on HTT capable architectures, the results of implementing -parallel- according
to the machines physical cores versus it logical's shows small though significant differences.

-parallel- is especially handy when it comes to implementing loop-based
simulation models (or simply loops), Stata commands such as reshape , or any job
that (a) can be repeated through data-blocks, and (b) routines that processes big
datasets (see the append section). Furthermore, the commands
-parallel bs- and -parallel sim- are
specially designed to easily implement bootstrapping and (monte carlo) simulations
in parallel fashion.

At this time -parallel- has been successfully tested in Windows, Unix
and MacOS for Stata versions 10 to 12.

After several tests, it has been proven that--thanks to how -parallel- has been
written--it is possible to use the algorithm in such a way that other techniques
of parallel computing can be implemented; such as Monte Carlo Simulations,
simultaneously running models, etc.. An extensive
example through Monte Carlo Simulations is provided
here.

4. Parallel Append

Imagine we have several dta files named -income.dta- stored in a set of folders
ranging 2008_01 up to 2012_12, this is, a total of 60 files (12 times 5) monthly ordered
which may look something like this:

2008_01/income.dta
2008_02/income.dta
2008_03/income.dta

...more files...

2010_01/income.dta
2010_02/income.dta
2010_03/income.dta

...more files...

2012_10/income.dta
2012_11/income.dta
2012_12/income.dta

Now, imagine that for each and every one of those files we would like to
execute the following program:

program def myprogram
gen female = gender == "female"
collapse (mean) income, by(female) fast
end

Instead of writing a forval/foreach loop (which would be the natural
solution for this situation), -parallel append- allows us to smoothly solve
this with the following command.

. parallel append, do(myprogram) prog(myprogram) ///
e("%g_%02.0f/income.dta, 2008/2012, 1/12")

Where element by element, we are telling parallel:

(1) do(myprogram): execute the command -myprogram-,
(1) prog(myprogram): -myprogram- is a user written program, and
(1) e("%g_%02.0f/income.dta, 2008/2012, 1/12"): this should process files 2008_01/income.dta up to 2012_12/income.dta.

Besides of the simplicity of its syntax, the advantage of using -parallel append-
lies in doing so in a parallel fashion, this is, instead of processing one file
at a time, -parallel- manages to process these files in groups of as
many files as clusters are set. Step-by-step, what this command does is:

1. Distribute groups of files across clusters

Once each cluster starts, for each dta file

2. Opens the file using [if] [in] accordingly to in and if options.

3. Executes the command/dofile specified by the user.

3. Stores the results in a temp dta file.

Finally, once all the files have been processed

4. Appends all the resulting files into a single one.

5. Caveats

When the -stata_cmd- or -do-file- saves results,
as -parallel- runs Stata in batch mode,
none of the results will be kept. This is also true for matrices,
scalars, mata objects, returns, or whatever
other object different from data.

Although -parallel- passes-through programs, macros
and mata objects, in the current version it is not capable of doing the same with
matrices or scalars.

6. Technical note

In order to protect a pll_id code (and thus ancillary files), once -parallel-
is called it creates a new file called __pll[pll_id]sandbox
(stored at c(tmpdir), in your case: ). This
forbids -parallel clean- from deleting any auxiliary file used by that process
and reserves the pll_id so that no other call of -parallel- can
use this pll_id. Once every child process has finished, the sandbox file
is removed, freeing the pll_id.

If for any reason the algorithm breaks due to a flaw or crush of the system,
the sandbox file and the rest of auxiliary files will not be deleted. In order
to clean up this, the user will be able to do so manually (moving the file(s) to
the OS recycle bin) or using parallel clean, all force
syntax. This way all sandbox files in the c(tmpdir) folder and auxiliary files
stored at the current directory will be deleted.

In earlier versions of -parallel-, tempfiles generation was
not safe as while running multiple Stata instances simultaneously these could
overwrite each other's tempfiles. Starting version 1.14, this is no longer a
problem as each Stata instance starts with a different c(tmpdir) location.
This way, instances' tempfile management will not interfere with each other, allowing
to safely use commands or algorithms depending on tempfile generation (such as
preserve and restore).

The option -setparallelid- is designed to let programmers recycle a
parallel id (pll_id). Intended to be used with -parallel_sandbox-
(undocumented, please refer to the source code of
-parallel_sandbox()-),
this option allows calling parallel several times using the same pll_id, which
makes auxiliary files management far simpler. Take the following example

program def mypllwrapper

// Reserving a pll_id
m: parallel_sandbox(5)

// Using the generated pll_id
save __pll`parallelid'_mypllwrapper, replace

// Recycling the pll_id
forval i=1/10 {
parallel, setparallelid(`parallelid') keep: some_other_cmd
}

// Cleanning up and freeing the pll_id. This will remove all files
// and folders named with prefix '__pll[parallelid]'
parallel clean, e(`parallelid')
m: parallel_sandbox(2,"`parallelid'")

end

For a real example of this, please
see -parallel.bs- and
-parallel_sim.ado-.

Example 1: using prefix syntax

In this example we'll generate a variable containing the maximum
blood-pressure measurement (bp) by patient.

Setup for a quad-core computer

. sysuse bplong.dta
. sort patient

. parallel setclusters 4

Computes the maximum of bp for each patient. We add the option by(patient)
to tell parallel not to split stories.

. parallel, by(patient): by patient: egen max_bp = max(bp)

Which is the ``parallel way'' to do:

. by patient: egen max_bp = max(bp)

Giving you the same result.

Example 2: using -parallel do- syntax

Another usage that may get big benefits from it is implementing loop-base
simulations. Imagine that we have a model that requires looping over each and
every record of a panel-data dataset.

Using -parallel-, the proper way to do this would be using the ``parallel do''
syntax

. use mybigpanel.dta, clear

. parallel setclusters 4
. parallel do mymodel.do

. collapse ...

where mymodel.do would look something like this

begin of do-file

local maxiter = _N
forval i = 1/`maxiter' {
/* Break key check (this line is optional) */
parallel break
...some routine...
}

end of the do-file

Or, in the case of using mata, this would look something like this

begin of do-file

mata:
N=c("N")
for(i = 1;i<=N;i++) {
/* Break key check (this line is optional) */
parallel_break()
...some routine...
}

end of the do-file

Example 3: setting the right path

In the case of -parallel- setting the stata.exe's path wrongly, using
-setstatapath- will correct the situation. So, if
"C:\Archivos de programa\Stata12/stata.exe" is the right path we only have
to write:

. parallel setclusters 2, s("C:\Archivos de programa\Stata12/stata.exe")

Example 4: Using -parallel bs-

In this example we'll evaluate a regression model using bootstrapping

Setup for a quad-core computer

. sysuse auto, clear

. parallel setclusters 4

Running parallel bs.

. parallel bs: reg price c.weig##c.weigh foreign rep

Which is the ``parallel way'' to do:

. bs: reg price c.weig##c.weigh foreign rep

Example 5: Using -parallel sim-

Example from simulate

Setup for a quad-core computer

. parallel setclusters 4

Experiment that will be performed

program define lnsim, rclass
version
syntax [, obs(integer 1) mu(real 0) sigma(real 1) ]
drop _all
set obs `obs'
tempvar z
gen `z' = exp(rnormal(`mu',`sigma'))
summarize `z'
return scalar mean = r(mean)
return scalar Var = r(Var)
end

Running parallel sim.

. parallel sim, expr(mean=r(mean) var=r(Var)) reps(10000): lnsim, obs(100)

Which is the ``parallel way'' to do:

. simulate mean=r(mean) var=r(Var), reps(10000): lnsim, obs(100)

Example 6: Using -pll_instance- and -PLL_CLUSTERS- macros

By using -pll_instance- and -PLL_CLUSTERS- global macros the
user can run -parallel- in such a way that each cluster performs a different
task. Take the following example:

Setup for a quad-core computer

. parallel setclusters 4
. sysuse auto, clear

program def myprog
gen x = $pll_instance
gen y = $PLL_CLUSTERS

// For the first cluster
if ($pll_instance == 1) gen z = exp(2)

// For the second cluster
else if ($pll_instance == 2) {
summ price
gen z = r(mean)
}

// For the third and fourth clusters
else gen z = 0
end

Running the program

. parallel, prog(myprog): myprog

Here, running with 4 cores, the program -myprog- performs different actions
depending on the value (number) of -pll_instance-. For those observation in the
first cluster, -parallel- will generate -z- equal to exp(2), for those in
the second cluster it will compute -z- equal to the average price and for the
rest of the clusters it will generate -z- equal to zero.

Example 7: Using -parallel- on Windows in batch-mode

Normally, the parallel instances of Stata are executed using -winexec-. This
command is ignored, however, on Windows when in batch-mode. A work-around for
this environment is to have Stata write out the commands to be executed to a file
(called the gateway) and have a separate process read new inputs to this file and
execute the commands. This latter part requires the user to install Cygwin and run
a few commands prior to starting Stata. In a Cygwin terminal, navigate to the appropriate
directory and do the following:

$ rm pll_gateway.sh
$ touch pll_gateway.sh
$ tail -f pll_gateway.sh | bash

Then you can execute your Stata script in batch-mode on Windows. The Cygwin tail
process can stay running through multiple uses.

The default gateway file assumed is pll_gateway.sh. If you would like a different
file modify the Cygwin script above and pass a new value for gateway_path.

Since Cygwin is going to execute the commands to start the parallel Stata instances
it needs a Cygwin-like Stata path. If the user does not specify the Stata path then
-parallel- will take the generated windows path and convert it to "/cygdrive/<drive letter>/...".
If this does not work you will need to specify the statapath explicitly.

8. Saved results

-parallel- saves the following in r():

Scalars

r(pll_n) Number of parallel clusters last used

r(pll_t_fini) Time took to appending and cleaning

r(pll_t_calc) Time took to complete the parallel job

r(pll_t_setu) Time took to setup (before the parallelization) and to finish the job (after the parallelization)

r(pll_t_reps) In the case of keeptime, the number of time measures performed.

r(pll_errs) Number of clusters which stopped with an error.

Macros

r(pll_id) Id of the last parallel instance executed (needed to use parallel clean)

r(pll_dir) Directory where parallel ran and stored the auxiliary files.

r(pll_seeds) Seeds used within each cluster.

-parallel bs- and -parallel sim- save the following in e():

Scalars

e(pll) 1.

-parallel version- saves the following in r():

Macros

r(pll_vers) Current version of the module.

-parallel- saves the following global macros:

$LAST_PLL_DIR A copy of r(pll_dir).

$LAST_PLL_N A copy of r(pll_n).

$LAST_PLL_ID A copy of r(pll_id).

$PLL_LASTRNG Number of times that -parallel_randomid()- has
been executed.

9. Development

You can always have access to the latest version of -parallel-. One option
is from its github repo (on-development source code):

https://github.com/gvegayon/parallel

Or from the project's website:

. net install parallel, from(https://raw.github.com/gvegayon/parallel/master/) replace
. mata mata mlib index

In the case of bug reporting, you can submit issues here:

https://github.com/gvegayon/parallel/issues

Please try to include steps to reproduce the issue and the output of the Stata
command -creturn list-.

10. mata source code

Most of -parallel- has been programmed in mata. This means that, as a difference
from typical ado files, -parallel- is distributed with lparallel mata library
(compiled code) and thus source code can not be reached directly by users. Given
this, the help file parallel_source.sthlp is included
in the package, help file which contains the source code in a fancy way.

In order to get access to different sections of the source code you can follow these
links:

Stops a cluster after the user pressed break parallel_break.mata
Remove auxiliary files parallel_clean.mata
Distributes observations across clusters parallel_divide_index.mata
Export global macros parallel_export_globals.mata
Export programs parallel_export_programs.mata
Wait until a cluster finishes parallel_finito.mata
(on development) parallel_for.mata
Normalize a filepath parallel_normalizepath.mata
Generate random alphanum parallel_randomid.mata
Lunch simultaneous Stata instances in batch mode parallel_run.mata
Set of tools to protect parallel aux files parallel_sandbox.mata
Set the number of clusters parallel_setclusters.mata
Set the Stata EXE directory parallel_setstatapath.mata
Write a ``diagnosis'' parallel_write_diagnosis.mata
Write a dofile to be paralellized parallel_write_do.mata

11. References

Luke Tierney, A. J. Rossini, Na Li and H. Sevcikova (2012). snow: Simple Network of Workstations. R package version 0.3-9. http://CRAN.R-project.org/package=snow

R Core Team (2012). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www.R-project.org/.

George Vega Y (2012). Introducing PARALLEL: Stata Module for Parallel Computing. Chilean Pension Supervisor, Santiago de Chile, URL http://fmwww.bc.edu/repec/bocode/p/parallel.pdf.

George Vega Y (2013). Introducing PARALLEL: Stata Module for Parallel Computing. Stata Conference 2013, New Orleans (USA), URL http://ideas.repec.org/p/boc/norl13/4.html.

Haahr, M. (2006). Random.org: True random number service. Random.org. http://www.random.org/clients/http/.

12. Author

George Vega Yon [cre,aut], California Institute of Technology. mailto:g.vegayon@gmail.com
http://www.its.caltech.edu/~gvegayon

Brian Quistorff [ctb], University of Maryland. mailto:bquistorff@gmail.com http://econweb.umd.edu/~quistorff/index.html

13. Contributors

Special Thanks to:
Elan P. Kugelmass (aka as epkugelmass at github) [ctb],
Timothy Mak (University of Hong Kong) (author of miparallel)

Damian C. Clarke (Oxford University, England),
Felix Villatoro (Superintendencia de Pensiones, Chile),
Eduardo Fajnzylber (Universidad Adolfo Ibáñez, Chile),
Eric Melse (CAREM, Netherlands),
Tomás Rau (Universidad Católica, Chile),
Research Division (Superindentendia de Pensiones, Chile),
attendees to the Stata conference 2013 (New Orleans),
Philippe Ruh (University of Zurich),
Michael Lacy (Colorado State).

14. Also see

Manual: [GSM] Advanced Stata usage (Mac),
[GSU] Advanced Stata usage (Unix),
[GSW] Advanced Stata usage (Windows)

Online: Running Stata batch-mode in Mac,
Unix and
Windows