help multishell v. 2.1 - July 2021

Title

multishell - allotting do files and variations of loops across parallel instances of Stata and computers efficiently.
Do files and loops are dissected and distributed across parallel instances of Stata and computers, mimicking a cluster.

See list of Contents and Description below.
multishell is a complex command, please read the help file carefully before using it.

Syntax

Setting up multishell

multishell path path [,clear]

Sets the path for temporary files and do files.
If option clear is used, multishell deletes ALL files and folders in the directory irrevocably!

multishell adopath path

Specifies path for ado files.
Necessary if multishell is not installed into one of the main adopath directory.

multishell alttext old text @ new text

string which is replaced from old text to new text in the do files specified by multishell add.
This option is useful if multishell is used to run do files across computers with a different folder structure or to insert computer specific commands.
For an example see Examples.
Number of text to be replaced is not limited (in theory).

multishell add path [,skiploop]

Adds do file to queue.
The number of do files is not restricted (in theory at least).
/* multishell loop*/ ensures that only certain loops are considered.
If option skiploop is invoked, all loops are skipped, meaning that to do file
as it is run.
Loops marked with /* multishell loop */ will be ignored.
See loops in multishell.

Optional

multishell exepath path

multishell exepath is optional, only necessary if multishell cannot find the Stata.exe.
The option is necessary to point multishell to the correct Stata.exe.
Under the default settings multishell automatically detects the Stata.exe.

Setting up the seed/random-number state

multishell seed save filename

Saves the used random-number state as a .dta in the folder defined by multishell path.
The state is saved together with the id and the variation of the loop of the task.
This option makes hardly sense as without further options, the states will be the same for all tasks.

multishell seed load filename [, state noseedstream]

Loads the seed or states from filename.dta.
The seeds have to be saved as variable seed and the number of the task in variable id.
The default is that what is saved in variable seed are seeds.
If they are random-number states (see rngstate), then option state needs to be used.
In this case set rngstate rather than set seed is used to set the random
number generator.

Option noseedstream has only an effect for Stata 15 or later.
If the option is invoked, all tasks will have the same seed-stream, namely stream 1.
Otherwise they will use the task number as seed stream, which is the default.

This option is useful if, the random-number state is saved by a earlier simulation and
results are meant to be reproduced.

multishell seed create filename [, seed(#|random) rngstate(rngcode) noseedstream]

A dummy dataset with seeds defined by seed() or rngstate() is created.
The dataset can be manipulated and then used to run multishell run, together with
multishell seed load.
The random-number states are saved in filename.
If none of the options is used, then no seed is set and Stata's initial seed is used.

With the option seed(#) an unique seed for all tasks can be set.
If seed(random) is used, then a random seed is used from random.org.
For Stata version before 15, a different seed for all tasks is obtained.
For Stata 15 and later, one seed is obtained and then each task is allocated a different stream.
In general, a use of Stata 14 or earlier is not recommended, see multishell seed.

Option noseedstream has only an effect for Stata 15 or later.
If the option is invoked, all tasks will have the same seed-stream, namely stream 1.
This implies that all random numbers are drawn from the same sequence.
The option is only recommended if this is on purpose.

It is recommended to use Stata 15 with mutishell.
When multishell seed save or multishell seed create is used with Stata 15,
multishell uses Stata's rngstream function (unless option noseedstream
is used), see multishell seed.

All seed options require multishell path and multishell add.
If multishell seed is not used Stata's initial seed applies.
This implies that the random number generator draws the same random numbers even if using different instances of Stata.
For further information about setting the seed in Stata, see set_seed.

For an overview of the options and their effect on random numbers see
overview of seed options.

mulitshell seed is only required for the server.

Starting multishell

multishell run [, threads(integer) sleep(integer)
nolog ncls
stop(date time [, killall]) maxtime(time)
continue
seedstream
]

Starts the server (or single computer).

Option threads(integer) defines the maximum number of parallel Stata instances.
It is not advisable to set more threads than number of cores of the CPU.

sleep(integer) sets the miliseconds multishell waits
until it refreshes the task list and starts a new instance.

nolog avoids saving a log file.
The default is that a log file is saved in the folder defined by path.
A clickable link to the log file will appear in the output.
Each log file starts with an overview, which includes the set seed, the random-number generator
state, the seed stream number and the random-number generator type.
In addition the name of the parent do file, the folder and the
current variation of the loop is show.

The default is that each time multishell is refreshed the output is cleared.
Option ncls avoids the output window to be cleared.

Options stop(date time [, killall]) and maxtime(time)
restrict the running time of multishell.
See Stopping multishell.
Option continue is only of use in conjunction with
stop() and maxtime().

seedstream is only available for Stata 15 and if multishell seed is not used.
seedstream uses the current random-number state and uses different seed streams
for all tasks. Using multishell seed save seeddta, rngstate(`c(rngstate)')
would lead to the same result.

multishell run client [, threads(integer) sleep(integer)
nolog ncls
stop(date time [, killall]) maxtime(time)]
continue
seedstream]

Starts the client.

Options threads(integer), sleep(integer),
nolog, ncls, stop(date time)}, seedstream and maxtime(time) are defined as above.
mutltishell add is not required as the client receives the tasks from the server.
nostop causes the client to continue running, even if all tasks are finished.
This option allows to start new tasks on client by starting a new multishell instance on the server.

Restart multishell

multishell restart type , computer(computername)

Restarts (or re-queues) the tasks defined in type for the computer with the name defined in the option computername.
This option is useful if a computer crashes, a do file aborts with an error and tasks are not completed.
multishell can be running on other computers, the task list will automatically be updated.

type can be:

type Description

assigned all assigned tasks to queued.
running all running tasks to queued.
finished all finished tasks to queued.
error all tasks with an error to queued.
stopped all tasks which were stopped to queued.
id(#) task number # to queued.

The changes apply only to the computer defined in computer(computername).
To restart all computers, use option computer(_all).
All types can be combined and called at the same time.

Diagnosis

multishell status

Displays the output window and an overview of all visible clients.
Lists additional set ado paths, path to exe file, temporary path and strings to be replaced by alttext.

Contents

Syntax

Description

Multishell and seeds

- Overview of seed options

Example 1: Single Computer

Example 2: Multiple Computers

Example 3: Continue and restart multishell

Example 4: Using multishell seed

Example 5: Skipping loops

Example 6: Stopping multishell

Example 7: Short Examples

Known Problems and Limitations

About

Disclaimer

Further See

Description

multishell allows the efficient processing of loops and multiple do files across a single and multiple computers.
multishell dissects forvalues and foreach loops and creates for each variation (tasks) a separate do file.
The do files are queued and sequentially processed.
Besides the do file, a .bat file is created for each task.
Then Stata's build in shell command is used to start a new instance of Stata using the .bat file.
The instance is closed as soon as the task is completed (or failed, then it is reported) and a new instance processing the next task is started.
One instance is reserved to organise the tasks and starts other instances.
Multiple instances can be run in parallel on the same computer.
Please read carefully the section about multishell and seed if running simulations which involve
drawing random numbers.

In addition multishell can use the computation power of multiple computers, mimicking a cluster.
The requirement is that all computers have access to the folder multishell uses.
The structure pointing to the folder, such as parent folders or drives is allowed to vary across computers.
Access can be either in form of a network drive or using an internet cloud based service such Dropbox or Backup and Sync from Google.
One computer acts as a server and assigns tasks to the clients.
As in the single computer case, one instance manages the incoming tasks and runs them.

The main aim of multishell is to make efficient use of computational power, of one machine or a network, to run simulations with multiple changing parameters.
For example, it is common to use Monte Carlo simulations to assess the bias of an estimator.
This is done by varying the number of observations, let's say from N=10 in steps of 10 to N = 130.
Assume the DGP and the regression are part of the program MonteCarloSim.
The number of observations is set as the only argument of the program and the estimated coefficient of variable x is returned as r(x).
This can be coded using Stata's simulate command nested in a forvalues loop:

forvalues n = 10 (10) 130 {

simulate bx = r(x), reps(1000) : MonteCarloSim `n'

}

multishell creates for each of the variations of n (n=10, n=40,...,n=130) a do file and a .bat file.
The files are then queued and consecutively processed by multiple instances of Stata on a single computer or by multiple computers.
The number of parallel Stata instances depends on the CPU and the number of cores.
If the machine has 8 cores, then 7 instances plus the instance managing the tasks can be started.
It is not advised to start more instances of Stata in total than the CPU has cores!
Windows usually automatically distributes the workload of a Stata instance to a different core.
For example the managing instance uses core 1, the first instance called by multishell core 2, and so on.

It is possible to process only certain loops and thus skip other loops.
If the differentiation is made, loops which are supposed to be dissected need to be
labelled with /* multishell loop */.
multishell automatically dectes if this option is used and adjusts accordingly.

If a task (or Stata instance) aborts with an error, multishell will continue and start the next task in the list.
The error will be posted in the output window.
In addition log files, saved as .txt, can be found in the folder path\temp\task, where path is the path specified by
multishell path and task is the task number.
The multishell server or client, or a Stata instance running a task can be stopped by using the break button or closing the Stata window.
It is possible to continue multishell and restart the server and client.
Options, such as the number of maximum Stata threads can be changed and further do files added.
If the server is stopped, no new tasks will assigned during this period, but remaining tasks will continue in the background.

multishell can stop tasks and itself after an user specified running time
or after a certain date and time is reached.
To do so multishell maintains a list with all running Stata instances
and their Windows Process ID.
If a tasks overruns, it automatically closes the running Stata instances.
Starting multishell with several Stata instances does not work however,
as the program cannot determine which Stata instance is the main instance.

multishell and seeds

Using multishell has important implications for the seed.
Please read carefully to ensure correct results.
There are two main difficulties when running parrallel isntances of Stata.

Same numbers across tasks and instances

The first is, that random numbers across instances (tasks) and computers can be the same.
Stata has a powerful random number generator, as described in set seed.
If not changed, the state of the random number generator (seed) is the same when Stata is started and therefore #
drawn numbers are identical.
If in a simulation random numbers a drawn and it is run on separate instances of
Stata on the same machine, the numbers will be the same.
The same applies if the same different tasks are performed on mutiple machines.
This implies that in the extreme, a Monte Carlo simulation using multiple
instances of Stata even on different computers produces the same results.
Stata uses a sequence of pre-determined random numbers, specified by the seed or
the state of the random number generator. This means, that the following code produces
the same numbers:

clear
set obs 10
set seed 123
x = rnormal()
set seed 123
y = rnormal()
list x y

The idea is to assign a unique seed to each task:

clear
set obs 10
set seed 123
x = rnormal()
set seed 456
y = rnormal()
list x y

The nubers in x and y are different.

Different Seeds but overlapping seed streams

The second difficulty is, that the seeds can overlap and therefore the drawn
random numbers repeat.
As explained, Stata uses pre-determined seeds.
The sequence of the seeds is called a seed stream, thus two sequences can
overlap.

Let's assume two tasks are performed, x1 in the first with 10 observations
and x2 for the second with 20 observations.
Using the same seed would lead to the same first 10 random numbers.
The following example will show this.

clear
set obs 20
set seed 123
gen x1 = rnormal() if _n <= 10
set seed 123
gen x2 = rnormal()
list x1 x2

For a Monte Carlo simulation, x1 and x2 are not independent anymore
and thus results using this example would be invalid.
A simple solution would be to use a different seed for both.
However as shown next, the two seeds can overlap and produce same numbers.

The following code will draw 10 numbers, save the seed, draw further 10 numbers for
variable x1 and then resets the seed and draws further 10 numbers for x2:

clear
set obs 20
set seed 1234
gen x1 = rnormal() if _n <= 10
local seed = c(rngstate)
replace x1 = rnormal() if _n>10
set seed `seed'
gen x2 = rnormal() if _n > 10
list x1 x2

We have shown that even though a different seed is set, the same random numbers
are drawn. Why is this the case?
The reason is that after each random number, the state of the random number generator
is set shifted along the seed stream.
The seeds at observation 11 overlap (i.e. they are the same) and
thus the same random numbers are produced.
The following graph (which resembles the one in rngstream) shows this.
For example, assume two tasks, first we draw 10 numbers, then 10 further numbers.

rng-State a b c
Task 1 |---------|---------|
Task 2 |---------|
Numbers 1 10 20

Task 1 starts with random number generator (rng) state a and draws 10 random numbers.
Then it reaches rng-state b and draws further 10 random numbers.
If for Task 2 the rng-state is set to b the 10 random numbers will be the same,
because the rng-states of both tasks overlap. This problem is more theoretical,
but it might occur and therefore is important to discuss.

Stata Version 14.2 and lower

For versions 14.2 and lower, a seed for each task is obtained from random.org.
This ensures that the seeds for each task will be different each time Stata is started.
However it is theoretically possible (and thus it will happen if repeated often enough, whatever
often means), that two or more sequences overlap each other.
Even though this is very unlikely, it is important for the user to note.
At the moment there is no workaround of this problem.

Seeds are saved if the options multishell seed save or
multishell seed create are used.

Stata Version 15 and following

For version 15 and following the two problems are sufficiently solved.
multishell seed create , seed(random) obtains a random number from
random.org which is used as a seed.
This ensures that if multishell run is repeated, respectively on different
computers, the initial seed differs (this is the first difficulty).

The second difficulty is solved by using seed streams
and is only available for Stata 15 following.
A different seed stream number is allocated to each task.
Following the example from above, each task (and thus Stata instance) is started
with the same intial seed obtained as a random number from random.org, however
each task is assigned a different stream number.
By construction, task 1 will be allocated stream 1, task two stream 2:

rng-State a1 b c a2
Task 1 |---------|---------|
Task 2 |---------|
Numbers 1 10 20 1 10

where a1 is the seed with stream 1 and a2 the same seed, but with stream 2.
With this method it is (almost) impossible to obtain the same sequence of random
numbers twice.
The method works across computers as long as the inital seeds are different.
For more background reading, please see set rngstream.

Seeds (the individual for each task) and the stream numbers are
saved if the options multishell seed save or
multishell seed create are used.

This function benefitted from many discussions at the London Stata User Group
Meeting 2018, especially with Ben Jann, Yulia Marchenko, Tim Morris, Austin Nichols and Adrian Sayers.

Overview of multishell's seed options

The following table shows the effect of the different options relating seeds and
random number-states. Three tasks are run, hence three instances of Stata and all
draw 100 random numbers. The seeds are shown in the tables.

For Stata 14:

multishell | seed save seed create dta, seed load seed
command | seed(random) seed(123)** not set

+

task 1 | default* random #1 123* seed #1** default*
task 2 | default* random #2 123* seed #2** default*
task 3 | default* random #3 123* seed #3** default*

+

where * means all seeds are the same across tasks,

** or corresponding rngstate(),

random #i, is random number i obtained from random.org,

seed #i is seed i in the dta with seed/random-number states.

For Stata 15 or later:

multishell | | seed create dta,
command | seed save | seed(random) | seed(123)**
| | seed stream | noseedstream | seed stream | noseedstream

+
+
+
+
+

task 1 | default* (1) random #1 (1) random #1 (1)* 123 (1) 123 (1)*
task 2 | default* (1) random #1 (2) random #1 (1)* 123 (2) 123 (1)*
task 3 | default* (1) random #1 (3) random #1 (1)* 123 (3) 123 (1)*

+

multishell | seed load dta, run, seed
command | seed stream noseedstream seedstream not set

+

task 1 | seed #1 (1) seed #1 (1) default (1) default (1)*
task 2 | seed #2 (2) seed #2 (1) default (2) default (1)*
task 3 | seed #3 (3) seed #3 (1) default (3) default (1)*

where the number of the random-number stream is parenthesis,

* means all seeds are the same across tasks

** or corresponding rngstate()

random #i, is random number i obtained from random.org,

seed #i is seed i in the dta with seed/random-number states,

seed stream is using the seed stream (no option required, default),

noseedstream means option noseedstream is used,

seedstream is the seedstream option of multishell run.

Example 1: Single Computer

Assume the Monte Carlo simulation example from above, using only one computer and allowing for 6 parallel running Stata instances.
To use multishell, two do files are required.
The first contains the code for the MonteCarlo simulation, the second one contains the code for multishell.
The simulation results are stored in a dataset called results_`n' in a separate subfolder.
The .ado files for multishell are located in multishell\ado in the documents folder.

The simulation including the program are saved in the do file MonteCarloSimulation.do in the subfolder multishell\simulation:

program define MonteCarloSim , rclass
syntax anything
clear
set obs `anything'
drawnorm x e
gen y = 1 + 0.5 * x + e
reg y x
return scalar x = _b[x]
end

clear
forvalues n = 10 (10) 130 {
simulate bx = r(x) , reps(1000) : MonteCarloSim `n'
save "C:\documents\multishell\results\results_`n'", replace
}

In the first part of the do file, the program for the Monte Carlo simulation is defined.
Within the program, the number of observations is set, random numbers for x and the error term e are drawn and y is calculated.
Then a simple regression of x on y is performed and the estimated coefficient on x is returned.
The second part of the do file includes the for-loop, which loops over different number of observations.
For programming issues, it is important that there is a space between the name of the local (here n) and the equal sign.
Within the loop, the MonteCarloSim program is called, executed 1000 times and the resulting coefficient estimates saved in a dta file called results.

The second do file contains the code for multishell.
The adopath for multishell is added, followed by the path for the temporary files and the Stata exe file.
The adopath option is only necessary if multishell is not loaded automatically with Stata.
In addition it allows the user to add further .ado files to Stata, which might be unique to the project.
Using multishell add the do file is added.
multishell creates the do files for each variation in a subfolder specified by path.
In the final line of code, the simulation is called and the running Stata instance calls up to 6 further threads.
The option sleep(2000) indicates that Stata waits for 2000ms before it refreshes the task list.

clear all
adopath ++ "C:\documents\multishell\ado\"
multishell path "C:\documents\multishell\test\output\", clear
multishell adopath "C:\documents\multishell\ado\"
multishell add "C:\documents\multishell\simulation\MonteCarloSimulation.do"
multishell run, threads(6) sleep(2000)

The resulting Stata window will look like:

----------------------------------------------------------------------------------------------------
# do-file State Time Machine
----------------------------------------------------------------------------------------------------
MonteCarloSimulation.do queued and running
1 n = 10 finished 17 Jul 2018 - 11:37:39 Research181
2 n = 20 finished 17 Jul 2018 - 11:37:39 Research181
3 n = 30 running 17 Jul 2018 - 11:37:29 Research181
4 n = 40 running 17 Jul 2018 - 11:37:29 Research181
5 n = 50 running 17 Jul 2018 - 11:37:29 Research181
6 n = 60 running 17 Jul 2018 - 11:37:29 Research181
7 n = 70 running 17 Jul 2018 - 11:37:27 Research181
8 n = 80 running 17 Jul 2018 - 11:37:27 Research181
9 n = 90 queued 17 Jul 2018 - 11:37:27
10 n = 100 queued 17 Jul 2018 - 11:37:27
11 n = 110 queued 17 Jul 2018 - 11:37:27
12 n = 120 queued 17 Jul 2018 - 11:37:28
13 n = 130 queued 17 Jul 2018 - 11:37:28
----------------------------------------------------------------------------------------------------
Machine Queued Assigned Running Finished Total
This Computer 5 0 6 2 8
----------------------------------------------------------------------------------------------------
Computername: Research181
as of 17 Jul 2018 - 11:37:39; started at 17 Jul 2018 - 11:37:28
next refresh in 2s.

The output shows the name of the do file, the number of tasks and a breakdown of all variations.
The State column indicates, weather a task is queued, assigned (only necessary for multi computer use, see below), running or finished.
The final columns indicate the time of the last action (i.e. queued, running, finished) and the name of the machine running the task.
Overview statistics are shown at the end.

As soon as a task is completed, stopped or aborts with an error, a clickable link
to the log file will appear, unless option nolog is used.
Each log file will start with an overview of the task.
The parent and running do file name and folder, variation and its number are shown.
In addition the random number type, the stream number, the current state of the random number generator
and the set seed are displayed as well.
The set seed is usually the seed which is inputted by multishell seed ... , seed(123) or
obtained from random.org.

The beginning of the log file looks the following:

******************************************************************************* *
* *
* Multishell Version 2.0 *
* *
* Parent File Properties *
* Folder: C:/Main Path/St~/test *
* do Filename: test.do *
* Running File Properties *
* Folder: C:/Main Path/St~/test/3 *
* do Filename: test_200.do *
* Variation No.: 3 *
* Variation: N = 200 *
* *
* Random Number Generator Properties *
* RNG Type : mt64s *
* Seed Stream Number : 3 *
* RNG State = XBA0000000027ede55c017af28fd825f52d5a6e0f4de27...... *
.....
* Set Seed = 669902172 *
* *
******************************************************************************* *

It is strongly advisable to carefully check the log file and especially the overview in the beginning with the seeds.

An example do file can be download from within Stata via net describe multishell, from(http://www.ditzen.net/Stata)
or net get multishell, from(http://www.ditzen.net/Stata).

multishell exepath is not required and will be omitted for the
remaining examples.

Example 2: Multiple Computers

For this example assume the Monte Carlo simulation from above is used and in addition a second do file for a panel estimation with fixed effects is added.
The code for the panel estimation is saved as MonteCarloSimulation_panel.do and reads:

program define MonteCarloSim , rclass
syntax anything
clear
tokenize `anything'
local n = `1'
local t = `2'
set obs `=`n*`t''
egen id = seq(), block(`t')
by id, sort: gen t = _n
xtset id t
drawnorm x e
gen y = 1 + 0.5 * x + e
xtreg y x, fe
return scalar x = _b[x]
end

clear
forvalues n = 50 (10) 130 {
forvalues t = 10 (10) 50 {
simulate bx = r(x) , reps(1000) : MonteCarloSim `n' `t'
save "PATH1\multishell\results\results_`n'_`t'", replace
}
}

The folder structure across computers can differ, for example different drives are used.
multishell allows for a computer specific structure in the do files as well.
PATH1 will be replaced by a string using the multishell alttext function.

A do file for each computer is required, as the server (Research181) and the client (HPJD) are called differently.
The file for the server is the same as above, but the second do file is added and the function alttext used.
In addition the number of instances (or threads) is restricted to 2.

clear all
adopath ++ "C:\documents\multishell\ado\"
multishell path "C:\documents\multishell\test\output\"
multishell adopath "C:\documents\multishell\ado\"
multishell alttext "PATH1 @ C:\documents\"
multishell add "C:\documents\multishell\simulation\MonteCarloSimulation.do"
multishell add "C:\documents\multishell\simulation\MonteCarloSimulation_panel.do"
multishell run, threads(6) sleep(2000)

Let's assume the folder structure for the client is the same, but on drive F, rather than C and in the subfolder Users.
The argument of the function multishell alttext is adjusted as well.
We allow 6 threads at the same time.
As the server will create the do files for the tasks, we do not need to specify those in the client do file.
Also note, the option clear for multishell path is not used.
It is only used for the client, which is started first:

clear all
adopath ++ "F:\users\documents\multishell\ado\"
multishell path "F:\users\documents\multishell\test\output\", clear
multishell adopath "F:\users\documents\multishell\ado\"
multishell alttext "PATH1 @ F:\users\documents\"
multishell run client, threads(2) sleep(2000)

As soon as we start the do file on the client, the Stata window is locked and the client searches in the directory for the Server.
The output window displays:

Check if multishell Server is set-up. . . . . . .

When the server is started and ready the following appears:

multishell Server set up, waiting for assigned tasks (for HPJD). . . .

As soon as the server assigned tasks to the client, the following is displayed in the output window:

----------------------------------------------------------------------------------------------------
# do-file State Time Machine
----------------------------------------------------------------------------------------------------
MonteCarloSimulation.do running and finished
1 n = 50 finished 17 Jul 2018 - 14:26:50 HPJD
2 n = 60 finished 17 Jul 2018 - 14:26:50 HPJD
3 n = 70 finished 17 Jul 2018 - 14:26:50 HPJD
4 n = 80 finished 17 Jul 2018 - 14:26:51 HPJD
5 n = 90 finished 17 Jul 2018 - 14:26:52 HPJD
6 n = 100 running 17 Jul 2018 - 14:26:50 HPJD
7 n = 110 finished 17 Jul 2018 - 14:26:41 Research181
8 n = 120 finished 17 Jul 2018 - 14:26:41 Research181
9 n = 130 finished 17 Jul 2018 - 14:26:50 Research181
MonteCarloSimulation_panel.do queued and running
10 n = 30 , t = 30 running 17 Jul 2018 - 14:26:43 Research181
11 n = 30 , t = 40 running 17 Jul 2018 - 14:26:53 HPJD
12 n = 30 , t = 50 assigned 17 Jul 2018 - 14:26:31 HPJD
13 n = 40 , t = 30 assigned 17 Jul 2018 - 14:26:32 HPJD
14 n = 40 , t = 40 running 17 Jul 2018 - 14:26:52 Research181
15 n = 40 , t = 50 assigned 17 Jul 2018 - 14:26:32 HPJD
16 n = 50 , t = 30 assigned 17 Jul 2018 - 14:26:32 HPJD
17 n = 50 , t = 40 queued 17 Jul 2018 - 14:26:33
18 n = 50 , t = 50 queued 17 Jul 2018 - 14:26:33
----------------------------------------------------------------------------------------------------
Machine Queued Assigned Running Finished Total
HPJD 0 4 2 5 11
This Computer 0 0 2 3 5
----------------------------------------------------------------------------------------------------
Total 2 4 4 8 16
----------------------------------------------------------------------------------------------------
Computername: Research181
as of 17 Jul 2018 - 14:26:54; started at 17 Jul 2018 - 14:26:33
next refresh in 2s.

The output shows the name of the two do files, which are consecutively processed, and the different tasks.
All tasks are distributed to both computers (the server, Research181 and the client HPJD).
There are four tasks which are assigned to the client, but are not yet started.

An example do file can be download from within Stata via net describe multishell, from(http://www.ditzen.net/Stata)
or net get multishell, from(http://www.ditzen.net/Stata).

Example 3: Continue and restart a running multishell

Continue a running multishell
It is possible to stop the server or the client clicking on the break button.
The tasks running in the background will continue until finished.
If the server is stopped, no new tasks will be assigned to the clients.
Both can be restarted using the multishell run command, for the client with the client extension.
The multishell set-up commands, path, adopath, alttext and if used exepath need to be executed again.
Do not use multishell add again, as the do files would be overwritten.

The option allows the change the number of parrallel Stata instances or the time to refresh.
Restarting the server using the code from above with 8 instead of 6 threads would read:

clear all
adopath ++ "C:\documents\multishell\ado\"
multishell path "C:\documents\multishell\test\output\"
multishell adopath "C:\documents\multishell\ado\"
multishell alttext "PATH1 @ C:\documents\"
multishell run, threads(8) sleep(2000)

The two lines containing multishell add were removed and threads(8) rather than threads(6) is used.
The remainder is the same as the code above.

Restart a running multishell

In the case a do-file aborts with an error, a Stata instance or a computer crashes, it is possible to continue multishell.
Using the example from above, assume computer HPJD crashed, all running and assigned tasks need to be resetted.
The set-up commands, path, adopath, alttext and if used exepath need to be executed again:

clear all
adopath ++ "C:\documents\multishell\ado\"
multishell path "C:\documents\multishell\test\output\"
multishell adopath "C:\documents\multishell\ado\"
multishell alttext "PATH1 @ F:\users\documents\"
multishell reset assigned running , computer(HPJD)
multishell status
multishell run client, threads(2) sleep(2000)

The command resets the status of the tasks which were assigned or running on computer HPJD.
multishell status displays the current state of multishell and the last line restarts the client.

If for example task 7 needs to be resetted, then the command line is:

multishell reset id(7), computer(Research181)

Example 4: Using multishell seed

The section about multishell and seed gives a basic
introduction and motivation for the use of multishell seed.
If multishell seed is not used, then no seed is set and different instances of Stata
might produce identical random numbers.
Before multishell seed can be used, a path has to be set and do files added.

Saving the seeds

In order to repeat a Monte Carlo simulation and to retrieve the same results, the state of the random number generator, the seed, is required.
multishell seed save filename saves the seed together with the id of the task and the variation. Following Example 1, one line of code is added before the final line:

clear all
adopath ++ "C:\documents\multishell\ado\"
multishell path "C:\documents\multishell\test\output\"
multishell adopath "C:\documents\multishell\ado\"
multishell add "C:\documents\multishell\simulation\MonteCarloSimulation.do"
multishell seed save "seed file"
multishell run, threads(6) sleep(2000)

The seeds of each task saved in the Stata dataset file seed file.dta.
The file contains the id of the task, the options (i.e. the value of n) and the seed.
For example the first two lines would read:

+----------------------------+
| id options seed |
|----------------------------|
1. | 1 n = 10 XN0000... |
2. | 2 n = 20 XN0000... |
3. ......

where the seed is abbreviated for a better readability.
As the seed was not specified further, it is the same for both variations and the first drawn numbers for both are the same.
It is now possible to use the saved seed from to reproduce the exact Monte Carlo simulation results, as shown next.

Using a saved seed

If the seed is already saved in the file seed file, multishell can open the file and assign it to the corresponding tasks.
The identification of the tasks is done by the task's id only.
To do so, the penultimate line is altered again:

clear all
adopath ++ "C:\documents\multishell\ado\"
multishell path "C:\documents\multishell\test\output\"
multishell adopath "C:\documents\multishell\ado\"
multishell add "C:\documents\multishell\simulation\MonteCarloSimulation.do"
multishell seed load "seed file"
multishell run, threads(6) sleep(2000)

For each of the variations, the seed is automatically loaded from the dataset and accordingly set.

Creating a dummy dataset

multishell can create a dummy dataset with empty seeds or prefilled "random" seeds.
The dataset contains a row for each variation, with the id and the variation itself.
It is possible to alter the dataset, change the seeds and the run multishell.
A dummy dataset is created using multishell seed create filename.
Let's assume the seed for the first variation is supposed to be 1, for the 2nd 2 and so on, the following code is required
(for a better readability for the following examples, the code starts after multishell add):

...
multishell seed create "seed file"
use "C:\documents\multishell\test\output\seed file", clear
replace seed = _n
save "C:\documents\multishell\test\output\seed file", replace
multishell seed load "seed file"
multishell run, threads(6) sleep(2000)

The dummy file is created and then opened. Then variable seed is replaced by _n, thus 1, 2, ..., N.
The dataset is saved, multishell loads the dataset and is started.

Using seeds from random.org

The above example would lead to different draws.
In addition it is possible to do the steps above, but with random numbers instead of _n, if the option seed(random) is used:

...
multishell seed create "seed file", seed(random)
multishell run, threads(6) sleep(2000)

multishell run can be called after , seed(random) without any further commands.
The random numbers are obtained from www.random.org and with a
altered version of setrngseed.
In the file seed file the seeds for all tasks are saved and can be used to replicate Monte Carlo simulation results.

Using the same seed

This option makes only sense for Stata 15 and together with the seed stream.

...
multishell seed create "seed file", seed(123)
multishell run, threads(6) sleep(2000)

In this case all Stata instances will have the seed 123.
If Stata 15 is used, each task will have a different stream and thus
the random numbers are unlikely to be repeated.

Example 5: Skip loops

As a default multishell processes all loops.
If some loops should be skipped by multishell and only certain considered,
then those to be processed can be marked with /*mulitshell loop*/.
For example, the same simulation as in multishell##example1 is run,
but the estimate for the intercept is saved as well.
The coefficients to be returned are saved in a local which is created by a loop:

program define MonteCarloSim , rclass
syntax anything
clear
set obs `anything'
drawnorm x e
gen y = 1 + 0.5 * x + e
reg y x
return scalar x = _b[x]
return scalar cons = _b[_cons]
end

clear
foreach coeff in x cons {
local ToReturn "`ToReturn' `coeff' = r(`coeff')"
}
/* mulitshell loop */ forvalues n = 10 (10) 130 {
simulate `ToReturn' , reps(1000) : MonteCarloSim `n'
save "C:\documents\multishell\results\results_`n'", replace
}

In the example above, multishell will skip over the
loop foreach coeff in x cons {.... } , but
will recognize the loop marked with /* multishell loop*/.
multishell will automaticall detect if any loop is marked and then only process those.
If no loop is marked, it will process all.

Thanks to Ben Jann for suggesting this option.

If option skiploops is used, then all loops are skipped.
The entire do file is run as it is in one instance of Stata.
This option is intended parallelzing large do files.

Example 6: Stopping multishell

Two options are available to restrict the running time of multishell.
For both it is important that the only one instance of Stata is running
when multishell is started.

Setting a specific date and time to stop multishell

multishell run , ... stop()

specifies a date and time multishell will stop.
For example, a simulation is set up and supposed to run over the weekend.
At Monday morning, say 1. October 2018 at 8.30am, it should stop, no matter if the
simulations are finished or not.
Then multishell can be started with the following parameters:

multishell run, ... stop("1 Oct 2018 8:30")

In this case mutlishell will stop on 1st Oct 2018 at 8:30 to start
new tasks.
If set up as a server, no new tasks for the client will be assigned.
Running tasks in the background will be completed.
To interrupt running tasks, the option killall can be used:

multishell run, ... stop("1 Oct 2018 8:30, killall")

In this case all runnings tasks will be closed.

Note: Date and time format have to be the same as stored in the macro
c(current_date), see c(current_date).
Time can only be defined by hours and minutes, seconds are omitted.
The option killall has to be enclosed in parenthesis.

Setting a maximum time for a task

It is possible to abort a task after a specifed amount of time.
The following option can be used when multishell ist started:

multishell run, ... maxtime(time)

where time is in the format hh:mm.
For example if a task is limited to run for 10 minutes:

multishell run, ... maxtime("0:10")

is used. Then any task is closed after 10 minutes.
Option maxtime() is computer specific.
This means, if it is used on a server, but not on a client, tasks on a client
are allowed to run longer. Same principle applies to the opposite case.

Technical Notes

a) If the options maxtime() and stopp() are invoked, multishell
maintains a computer specific list of running Stata instances and their
Windows process id.
The list is obtained from the command line of the operating system.
The process id is then used to stop a specific task.
When multishell is started, only one instance of Stata is allowed to run.
Otherwise it will get confused of which instance is the main instance.
If further Stata instances are started, for example to do some other
work, multishell can get confused if the additional
instance is started between a start of a new multishell Stata instance
and a refresh.
This is more a hypothetical case, but can occur.

b) It is possible to re-start a multishell server or client, if it is done
from the same Stata instance.
In this case, option continue needs to be used.

c) Server and client are independent and
stop() and maxtime()
need to be set on both machines.
The server cannot kill a task on any client or vice versa.

Thanks to Mark Schaffer and Adrian Sayers for the idea and discussions
of this feature.

Example 7: Short Examples

The log files are saved in the subfolders for each variation in Z:\test.

nostop option

The nostop option is only available for the client.
multishell will keep running after all variations are worked through.
The option allows to finish one task and then start a new list of tasks from the server.
For example, the client is started

....
multishell run client, threads(2) sleep(2000) nostop

and the do file first simulation is started from the server with:

....
multishell add "C:\documents\first simulation.do"
multishell run, threads(2) sleep(2000)

The file is processed and the simulations are finished.
Then the multishell server stops.
The multishell client instance keeps running and awaits the next task.
If a new multishell instance is started on the server, like

....
multishell add "C:\documents\second simulation.do"
multishell run, threads(2) sleep(2000)

the client will pick it up and start running variations.
Thus, it is possible to start new instances of Stata on the client without being present in front of the actual computer.

Known Problems and Limitations

- In general multishell has sometimes problem with mutliple Stata versions accessing the same files.
Errors are often caused by slow internet or network connections.
Restarting the do file usually helps.

- Seeds, to obtain seeds from www.random.org an internet connection
is necessary.
If multishell seed is used across computers with different versions
of Stata (i.e. 14 and 15) problems with the seeds might occur. In general, it is
advisable to use the same version of Stata for all computers.

- Special characters in do files such as single quotes (` and '), double quotes ("),
can produce an error message which can be ignored.

- Loops multishell tries to dissect all loops, even if they are not part of the simulation or commented out (by a "*").
Best is to move all other loops into a separate do file and include it by using the Stata command include.
Alternatively, mark only needed loops by /*multishell loop*/,
see Skipping loops.

- locals in loops at the moment it is not possible to use locals in the loops.
For example the following does not work:

foreach arg in `list' ...

The local `list' is unknown to the Stata instance processing the foreach loop and therefore the loop would be empty.

- Missing spaces between the name of the local and the equal sign lead lead to an error.
For example foreach n=1(1)10 does not work.
It is necessary to add a space between the name of the local and the equal sign, such as
foreach n = 1(1)10.

- Cloud services such as Dropbox and Backup and Sync from Google can be slow synchronizing files.
Read/write errors are usually not common, but can occur.

- Read/write errors occur if a file is not correctly closed. Best is to restart multishell.

- Only Windows is supported.

- If multishell is run on a mapped network drive, the log file is saved in the My documents folder.
This is a Windows/DOS bug.

- set more needs to be set off.

- multishell only speed ups loops.
It does not help to speed up existing Stata commands such as regress, simulate, etc.

Author

Jan Ditzen (Free University of Bozen-Bolzano)

Email: jan.ditzen@unibz.it

Web: www.jan.ditzen.net

Thanks to Francesca Watson for reminding me using batch files to run multiple instances of Stata.
multishell was presented at the Stata User Group Meeting 2018 in London and
I am greatful for many comments, especially (in alphabetical order) from
Ben Jann, Yulia Marchenko, Tim Morris, Austin Nichols, Adrian Sayers and Mark Schaffer.
The program code to obtain random numbers from random.org is taken from
setrngseed, written by William Gould and Antoine Terracol.

Please cite as follows:

Ditzen, J. 2021. multishell

The latest versions and examples can be obtained via net from http://www.ditzen.net/Stata.

Disclaimer

multishell is a user-written command and comes with no warranty intended or implied.
Please note that some of the functions involve deleting (sub)folders and files, so carefully check the folder before executing the command.
I do not take over responsibility for any computer crashes, lost work or financial losses following the use of multishell.

Changelog

This version: 2.1 - 29. July 2021

Also see

See also: parallel, qsub, setrngseed