*! version 1.0 02/10/2015 Adrian Sayers
help qsub

Title

qsub -- queue submission program to emulate a cluster environment using your desktop PC.

Syntax

qsub , jobdir(string)
[ maxproc(integer)
queuesheet(string)
flagstart(string)
flagend(string)
wait(integer)
waitb4append(integer)
statadir(string)
append
outputdir(string)
outputsavename(string)
deletelogs
]

Description

qsub is a queue submission program that compiles a lists of jobs in a folder and then submits them to different instances of Stata on a desktop computer, and then compiles results if requested.

The program provides a simple way to perform simulation studies, bootstrapping, multiple imputation, data level parallelisation operations and other computationally intensive tasks on a desktop PC utilising all available CPUs. It is also a useful way to debug programs before uploading them to a cluster environment.

Prior to using qsub, you are required to create a directory containing all the jobs you intend to implement, and optionally a folder to store results or data within.

By default qsub creates .log file for each instance of every evoked stata session, and records the output. These logs are excellent way of debugging mistakes and errors.

Compulsory Options

jobdir(string) Specifies the directory of stored job files.

Optional Options

maxproc(integer) Specifies the number of processors available for use on your desktop machine, default is 2.

queuesheet(string) Specify the name of the queuesheet of jobs to be completed, default is "queuesheet.dta".

flagstart(string) Specifies the number of processors available for use on your desktop machine, default is "flagstart.dta".

flagend(string) Specifies the name of the of processors available for use on your desktop machine, default is "flagend.dta".

wait(integer) Specifies the length of pause in seconds between looking for completed jobs, default is 1. Longer pauses should be used for jobs that take a long time, and short pauses for faster jobs.

waitb4append(integer) Specifies the length of pause before appending results, default is 1. Longer pauses should be used for slow systems with long lags when reading writing data.

statadir(string) Specifies the directory of Stata, default is "C:\Program Files (x86)\Stata14\StataMP-64.exe". However this is simply changed by editing the .ado file if required.

append Instructs Stata to append outputs into a single file.

outputdir(string)} Specifies where outputed results will be stored.

outputsavename(string) Specify the name of the appended files.

deletelogs Instructs Stata to delete *.log files created during the batch process.

How qsub works

1. qsub compiles a list of jobs saved in jobdir(string) and saves them into a file which is named using queuesheet(string) suboption, if no name is specified the default file name "queuesheet.dta" is created.

2. qsub then uses the queuesheet to assign jobs to an instance of stata based on the maximum number of processors indicated to be available for use using the maxproc(integer) option.

3. qsub appends a few lines of codes at the beginning and end of each do file to create indicator files to show when a job has started and when a job has ended. The indicator files are named using the flagstart(string) and flagend(string) suboptions. If no options are specified the default names are used.

4. qsub assigns jobs based to a Stata instance based on the indicated number of processors available maxproc(integer), and reassigns jobs when the process has finished.

5. qsub then waits untill all jobs are finished before control is returned to Stata.

Optionally

6. qsub can compile outputs saved in a specific directory outputdir(string) and saves them in a single file specified by outputsavename(string).

7. qsub then deletes all log files "*.log" in the working directory, log files are automatically created when running Stata in batch mode from the working directory.

Example: Simulation

* Clear and then create a job, output and data directory.

* !WARNING! ! rmdir "folder" /s /q forcefully removes all content, useful for quick clears when debugging but be careful. !WARNING!

mkdir _output

mkdir _queue

mkdir _data

* Create your simulated data of interest

set seed 1

clear

set obs 50

gen x = 100*uniform()

gen y = 10 + 0.1*x

save _data/design.dta , replace

* Write a loop which creates a jobfile which echoes a do using mysimfile with arguments.

forvalues sim = 1/10 {

file open mydofile using _queue\sim`sim'.do, write replace

file write mydofile `"do mysimfile.do `sim' "'

file close mydofile

}

* Where mysimfile.do contains your code of interest for example:

* ----mysimfile.do----------------------------------------------------------------

* postfile results sim cons cons_se x x_se using _output/results`1'.dta, replace

* use _data/design.dta , clear

* set rngstream `1'

* set seed 1

* gen e =15*invnorm(uniform())

* gen yobs = y + e

* reg yobs x

* post results (`1') (_b[_cons]) (_se[_cons]) (_b[x]) (_se[x])

* postclose results

* --------------------------------------------------------------------------------

qsub , jobdir(_queue) maxproc(8) append deletelogs outputdir(_output) outputsavename(simresults.dta)

Example: Bootstrap

* Clear and then create a job, output and data directory.

* !WARNING! ! rmdir "folder" /s /q forcefully removes all content, useful for quick clears when debugging but be careful. !WARNING!

mkdir _output

mkdir _queue

mkdir _data

* Write a loop which creates a jobfile which echoes a do using mybootfile with arguments.

forvalues bs = 1/10 {

file open mydofile using _queue\boot`bs'.do, write replace

file write mydofile `"do mybootfile.do `bs' "'

file close mydofile

}

* Where mybootfile.do contains your code of interest for example:

* ----mybootfile.do----------------------------------------------------------------

* postfile results i proportion using _output/boot`1'.dta, replace

* webuse bsample1 ,clear

* set rngstream `1'

* set seed 1

* bsample

* ci proportion female , wald

* post results (`1') (`r(proportion)')

* postclose results

* --------------------------------------------------------------------------------

qsub , jobdir(_queue) maxproc(8) append deletelogs outputdir(_output) outputsavename(bootresults.dta)

Example: Multiple Imputation

* Clear and then create a job, output and data directory.

* !WARNING! ! rmdir "folder" /s /q forcefully removes all content, useful for quick clears when debugging but be careful. !WARNING!

mkdir _output

mkdir _queue

* Load some pretend data & remove mi set

webuse mheart8s0 , clear

mi unset

drop mi_*

gen id = _n

save mheart8s0 , replace

* Write a loop which creates a jobfile which echoes a do using mymifile with arguments.

forvalues mi = 1/10 {

file open mydofile using _queue\mi`mi'.do, write replace

file write mydofile `"do mymifile.do `mi' "'

file close mydofile

}

qsub , jobdir(_queue) maxproc(4) deletelogs

* Analyse imputed datasets

use mheart8s0 , clear

cd _output

mi import flongsep myimp , using( _1_imp{1-10} ) id(id) clear imputed( age bmi)

mi estimate : logit smokes age bmi , or

* ----mymifile.do----------------------------------------------------------------

* Load original data & mi set

use mheart8s0 , clear

* Change Directory to _output

cd _output

* Seperate Imputation locations to avoid conflicts in temporary files & change directory

mkdir imp`1'

cd imp`1'

mi set flongsep imp`1'

set rngstream `1'

set seed 1

* Specify Imputation

mi register imputed bmi age

mi impute chained (regress) bmi age = attack smokes hsgrad female, add(1)

* Remove mi indicators, and pretend its from an external source

use _1_imp`1' , clear

drop _mi_id

save _1_imp`1' , replace

* Erase mi set copy of original

erase imp`1'.dta

* Move Imputed to dataset to central location and tidy up

copy imp`1'\_1_imp`1'.dta _1_imp`1'.dta , replace

erase imp`1'\_1_imp`1'.dta

rmdir imp`1'

* Return to root folder

cd ..

* --------------------------------------------------------------------------------

Example: Data Level Parallelisation

* Clear and then create an output and job directory.

* !WARNING! ! rmdir "folder" /s /q forcefully removes all content, useful for quick clears when debugging but be careful. !WARNING!

mkdir _output

mkdir _queue

* Create some source data.

set seed 1

clear

set obs 200000

gen myid = round(50* uniform(),1.0)

save mylist.dta , replace

* Read in your source data and create a list of variables to parallelise your code over.

use mylist.dta , clear

levelsof myid , local(myidlist)

* Write a loop which creates a jobfile which echoes a do using mydofile with arguments.

foreach id in `myidlist' {

file open mydofile using _queue\job_`id'.do, write replace

file write mydofile `"do mydofile.do `id' "'

file close mydofile

}

* Where mydofile.do contains your code of interest for example:

*---mydofile.do--------------------------------------

* sysuse mylist.dta ,clear

* keep if myid==`1'

* egen count = count(myid)

* save _output\myid`1' , replace

*----------------------------------------------------

qsub , jobdir(_queue) maxproc(8) append deletelogs outputdir(_output) outputsavename(compiledresults.dta)

TopTip

Mount a RAM DISK. This speeds up the reads and writes of datasets and makes everything run more smoothly.

Remarks

qsub is deliberately written as a light program with a very specific focus. I hope the focussed nature will allow greater flexibility given the wide variety of potential uses.

qsub also does not asign an instance of Stata to a specific processor. qsub spawns instances and hopes Windows will assign them to processors in a sensible way.

Disclaimer

qsub comes with no warranty intended or implied. We recommend that users check their results with those obtained through other algorithms. Users are also encouraged to check their results with those produced by other statistical software.

Author

Adrian Sayers

University of Bristol

adrian.sayers@bristol.ac.uk

Acknowledgements

I would like to thank Nick Cox for allowing me to incorporate the filei command which is used to write lines of code at the beginning of the do file. I am also grateful to Tim Morris for testing qsub and providing helpful comments.

Also see

parallel set rngstream mirubin