*! version 2.0.0 04feb2022

batcher

batcher -- Simple task paralleliser for Stata dofiles (Windows only)

Syntax

batcher filename
, iter(numlist) tempfolder(string) options

Main
iter(numlist) iterations you want to execute

tempfolder(string) folder to store logs (used to track progress)

sts(string) sendtoslack url or saved name to get updates on success/failure

trackdelay(integer) seconds between start of dofiles and completion tracking

betweendelay(integer) seconds between start of each dofile

updatedelay(integer) seconds between updates of the tracker

maxparallel(integer) maximum concurrent instances

notrack disable progress tracker

sts_exceptsuccess only send message to slack on failure

nostop do not stop dofile on error (see do manual entry [PDF])

noquit keep Stata windows of iterations open after completion

notimestamp do not print timestamp on iteration start

saveoptions(string) saves specified options as default (so you can omit them)

stataexe(string) path to the Stata exe, normally not required

Deprecated
sleepduration(integer) seconds between start of dofiles and completion tracking

Description

batcher is a simple way to parallelise tasks in Stata (Windows only at the moment)

1. You specify which dofile to run and add an iteration number

2. batcher starts the dofile, feeds in the iteration number

3. The code in the dofile takes this iteration number to determine what to do

4. The main Stata window tracks progress by reading log files

5. Optionally send completion/failure updates to your phone through sendtoslack

The main inspiration for this command was frustration with existing parallelisation programs. They were either too complicated to use
or somehow failed to provide the results I needed. Instead, batcher is very straightforward to use and only requires minor changes
to the underlying dofile. There is no magic under the hood (except maybe the progress tracker). The code in your dofile determines
how the parallelisation happens, where everything is saved, etc. This program simply saves you the effort of opening multiple Stata windows
and fixing the setting each time.

Finally, to cater to my own laziness, batcher has a saveoptions feature, which places any options you specified
in the profile.do file. The command will then use these options the next time you use batcher,
without requiring you to add the options (overwrite by typing "overwrite", see examples).
It's quite cool, even though I say so myself.

Options

iter(numlist) This is the key of the whole program. Provide a numlist here, each element of which will be fed to your dofile as first argument.
See the examples for more info.

tempfolder(string) This option is required. You can simply point this to your Stata working directory, or to some Dropbox folder, it really
does not matter. I recommend saving this option through saveoptions (see later) so you only have to specify it once.

sts(string) If you specify a valid sendtoslack url or name in sts(), batcher will send a message to the specified Slack room (-> your smartphone) to inform you
upon errors or completion. If you want to send to the default you saved in sendtoslack, specify url(default). See sendtoslack for more information.

trackdelay(string) Personal preference. By default batcher waits 60 seconds to start the tracker, but on slow PCs this might cause issues, while for short programs this might be too long. The sleep duration is defined in seconds. Old name: sleepduration(string).

betweendelay(string) By default batcher waits 10 seconds before starting the next dofile, but on slower PCs or with many iterations this might cause issues. Then you should increase this delay. Defined in seconds.

updatedelay(string) By default the tracker updates every 30 seconds, which might not work for your setting. Defined in seconds.

maxparallel(string) Limits the number of concurrent instances. If your number of iterations I exceeds maxparallel M, then batcher will run M instances and start the remaining I-M consecutively as instances finish. If you are using Stata-MP, then maxparallel is automatically set to the number of (logical) processors you have. You can overwrite this behaviour by setting maxparallel to 0. The underlying idea is that running more instances than you have processors generally causes instability and speed reductions.

notimestamp Stop the program from printing timestamps as it starts instances.

notrack I don't know why you'd want to disable the tracker, but with this option you can.

sts_exceptsuccess Only meaningful if specified with sts(). Prevents the program from sending messages to slack on successes. Mainly relevant
if you are running many iterations and don't want to get spammed.

nostop This is a dofile option. Prevents the dofile from stopping on errors.

noquit By default, batcher closes the windows it opens to run the iterations. Specify noquit to keep them open.

stataexe(string) You shouldn't need to use this option as batcher uses internal machinery to detect the location of the Stata executable.
But if for some reason that would fail, you could manually specify it here.

saveoptions(string) Some of these options you might want to use all the time (e.g. you generally only use one sendtoslack url for the smartphone updates).
If you execute batcher with saveoptions(string), it will save any specified options to profile.do.
Any subsequent use of batcher will then use these options. In exceptions, you can specify that option with an "overwrite" argument.
That will not work for argument-free options. In that case you will have to clear the global yourself (e.g. global batcher_run "").
The examples might clarify this a bit.

Examples

Showing examples for batcher is somewhat tricky, as it requires a dofile to run.
We provided an example dofile to illustrate
some of the possibilities.
This dofile was saved in the same folder as the ado file.
Execute the following code to gain access to the location
of this dofile.

. qui findfile batcher.ado

. global examplePath = subinstr("`r(fn)'", "batcher.ado", "batcher_exampleDofile.do", .)

The code below opens this dofile, so you can use it as inspiration for your own projects.

. doed `"$examplePath"'

Let's try a first parallelisation exercise.

. batcher $examplePath, i(1/2) tempfolder(`c(pwd)')

Normally, this opened two stata windows. In the first it mentioned displaying 1, in the second displaying two.

This is because we first fed "1" to the dofile, then "2". You would find a log of the two runs in your current working directory (c(pwd)).

Next, we want to be informed on progress in Slack.

. batcher $examplePath, i(3/4) tempfolder(`c(pwd)') sts(https://hooks.slack.com/services/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)

This performed two regressions, with different independent variables in each iteration. It also sent messages to the webhooks channel

when each iteration finished and at the end.

Now let's see what happens when an iteration fails

. batcher $examplePath, i(5/7) tempfolder(`c(pwd)') sts(https://hooks.slack.com/services/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) sts_exceptsuccess

This attempted to calculate three means of prices. One for domestic cars, one for foreign cars and one for the non-existent category 999.

The first two ran without issue. But the third ended with a failure. We only received a message in slack for this failed iteration because

we specified the sts_exceptsuccess option. At the end of the batcher run, we were again informed that this iteration failed.

Now let's save the tempfolder option specified

. batcher $examplePath, saveoptions(tempfolder) i(1/2) tempfolder(`c(pwd)')

If you didn't have a profile.do yet, it created this file for you. Then it added one line to it which saves the tempfolder option.

Every time you start Stata, this global is defined and batcher knows where to find it. As a result, you no longer need to specify the option,

which we illustrate below.

Use saved option

. batcher $examplePath, i(1/2)

batcher used the tempfolder you saved earlier, without requiring you to specify the option.

Author

Jesse Wursten
Faculty of Economics and Business
KU Leuven
jesse (dot) wursten (at) kuleuven (dot) be}

Other commands by the same author

sendtoslack Stata Module to send notifications from Stata to your smartphone through Slack

stop Stata command to interrupt dofiles intelligently (closes logfiles and optionally sends a message to your smartphone)

xtqptest Bias-corrected LM-based test for panel serial correlation

xthrtest Heteroskedasticity-robust HR-test for first order panel serial correlation

xtistest Portmanteau test for panel serial correlation

xtcdf CD-test for cross-sectional dependence

timeit Easy to use single line version of timer on/off, supports incremental timing

pwcorrf Faster version of pwcorr, with builtin reshape option

cdo Alternative to Stata's do: get an update when the dofile stalls and other quality of life improvements