*! version 3.2 17oct2021

Title

pc_simulate -- Power calculations by simulation, using existing dataset

Syntax

pc_simulate depvar [if] [in] , model(pc_simulate##description:model) mde(numlist)
[options]

Required [+]
model(pc_simulate##description:model) type of model to be estimated; must be one of
ONESHOT, POST, DD, ANCOVA.

mde(numlist) list of minimum detectable effect sizes for calculating statistical power, in units of depvar (not in percent of depvar);
e.g. mde(10 20 30).

Experimental design parameters [+]
i(panelvar) cross-sectional unit of randomization, required for POST, DD, and ANCOVA models; must be numeric;
will default to stored r(panelvar) if not specified.

t(timevar) time period variable, required for POST, DD, and ANCOVA models; must be numeric;
will default to stored r(timevar) if not specified.

n(numlist) list of sample sizes for experiment (the number of cross-sectional units included in the randomization);
must be positive integer(s), e.g. n(50(10)80); default is # of unique values of panelvar.

p(numlist) list of treatment ratios (the proportion of cross-sectional units to receive treatment); must be between 0 and 1,
e.g. p(0.5 0.75); default is p(0.5).

pre(numlist) list of number of pre-treatment periods for DD and ANCOVA models; must be positive integer(s),
e.g. pre(2(2)10); default is pre(1).

post(numlist) list of number of post-treatment periods for POST, DD, and ANCOVA models;
must be positive integer(s), e.g. post(1 2 3);
defaults are post(2) for POST model, and post(1) for DD and ANCOVA models.

alpha(#) desired Type-I error rate (or false discovery rate); must be between 0 and 1; default is alpha(0.05).

onesided assumes one-sided hypothesis test, with the direction of the test determined by the sign of each mde(numlist) value;
if not specified, default is a two-sided hypothesis test.

Model options [+]
absorb(varlist) fixed effects to be included in estimating equation; must be numeric; will default to
absorb(panelvar timevar) for DD model only.

controls(varlist) linear control variables to be included in estimating equation; must be numeric.

vce(vcetype) vcetype may be unadjusted, robust or
cluster fvvarlist (see reghdfe for detail);

default is vce(unadjusted) for ONESHOT model, DD model with 1 pre-treatment and 1 post-treatment period,
and ANCOVA model with 1 post-treatment period;

default is vce(cluster panelvar) for POST model, DD model with (pre+post)>2, and ANCOVA model with post>1.

Simulation and output options [+]
bootstrap samples cross-sectional units with replacement;
accommodates sample sizes larger than # of units in the existing dataset; default is to sample
units without replacement (if bootstrap is
not specified).

nsim(#) number of simulations performed for each set of experimental design parameters; default is nsim(500).

outfile(filename) .csv file that stores simulation results in current directory; if not specified, filename will be automatically generated.

[append|replace] append adds simulation results to an existing .csv file; replace overwrites existing .csv file (if append is
not also specified).

Design options [+]
collapse estimates a collapsed regression model, with a single cross-sectional outcome averaged across all (pre- and) post-treatment periods;
this yields a cross-sectional regression for POST and ANCOVA models, and a two-period panel regression for a DD model.

stratify(varlist) stratifies randomization by a (list of) categorical covariate(s); if specified, then n(numlist)
governs the number of units in each stratified randomization cell; note that the resulting power calculations still estimate a model
with a pooled average treatment effect; cannot be combined with idcluster(groupvar).

idcluster(groupvar) defines a cross-sectional group identifier, in order to accommodate cluster-randomization; if specified,
simulations randomize at the cluster (i.e groupvar) level, and n(numlist) governs the number of clusters; panelvar units must
nest within groupvar clusters; cannot be combined with stratify(varlist).

sizecluster(#) defines the number of panelvar units within each groupvar cluster; must be combined with idcluster(groupvar);
default is size of each cluster in existing dataset.

pcluster(numlist) defines the treatment intensity within each groupvar cluster; must be between 0 and 1; must be combined with idcluster(groupvar);
multiple values yield heterogeneous treatment intensities, in equal proportions across treated clusters; default is pcluster(1).

Advanced options [+]
tstart(# [#]) restricts the range of time periods over which to begin experiment; for example, tstart(4 7)
will simulate experiments beginning in time periods timevar={4,5,6,7} with equal probability;
note that this indicates the first pre-treatment period in a DD or ANCOVA model.

absorbfactor(fvvarlist) accommodates factor variable fixed effects, as allowed by reghdfe;
can be combined with the simpler absorb(varlist) option.

weight(weight) fweights, aweights and pweights are allowed; syntax is weight(aw=varname);
see weight.

reghdfeoptions(options) allows users to pass through additional reghdfe options
(not all of which are supported); see reghdfe.

Description

This program performs power calculations by simulating a randomized experiment using an existing dataset. For each iteration,
it randomly assigns units to treated and control groups, imposes an average treatment effect on treated units,
estimates this treatment effect using a regression model,
and records whether the null hypothesis of zero treatment effects is
rejected at the chosen significance level.

This program accommodates four types of experiments:

- ONESHOT, with exactly 1 observation of post-treatment data
for each cross-sectional unit, and 0 pre-treatment observations

- POST, with 2+ observations of post-treatment data
for each cross-sectional unit, and 0 pre-treatment observations

- DD (difference-in-differences), with 1+ observations of both pre-treatment and post-treatment data
for each cross-sectional unit;
estimated as a two-dimensional panel regression with pre-treatment and post-treatment time periods for each unit

- ANCOVA, with 1+ observations of both pre-treatment and post-treatment data
for each cross-sectional unit;
estimated using post-treatment observations only, and including the pre-treatment average of the outcome variable as a linear control

For each model, the default specification is as follows (where D is the treatment indicator):

- ONESHOT: Y_i = b*D_i + e_i

- POST: Y_it = b*D_i + e_it

- DD: Y_it = b*D_it + fe_i + fe_t + e_it

- ANCOVA: Y_it = b*D_i + Ypre_i + e_it

For each iteration, the program randomly assigns units as either treated (D_it=1) or control (D_it=0). It then constructs the outcome variable
Y_it = depvar_it + mde*D_it.

For a (default) two-sided test, the model rejects the null hypothesis if b is statistically different from zero, given significance level alpha.

The average rejection rate across all nsim simulations is the statistical power, or the probability
that the experiment would reject a true effect size of at least mde.

Researchers typically design experiments to achieve 80% power, or power=0.8.
However, this value is arbitrary.

Options

Experimental design parameters

i(panelvar) assigns the cross-sectional unit identifier. This
is required for POST, DD, and ANCOVA models, which assume 2-dimensional panel datasets. For
ONESHOT model, i(panelvar) is still necessary if the dataset contains 2 (or more) dimensions. If
not specified for a ONESHOT model, the program assumes that each row of the dataset represents a separate cross-sectional unit.

t(timevar) assigns the time-period identifier. This
is required for POST, DD, and ANCOVA models, which assume 2-dimensional panel datasets. For
ONESHOT model, t(timevar) is still necessary if the dataset contains 2 (or more) dimensions.

n(numlist) governs the sample size of the experiment, or the number of cross-sectional units to be included in the randomization. If
not specified, n defaults to the number of unique values of panelvar in the dataset.

If n is less than the number of cross-sectional units in the dataset and the option bootstrap is not specified, then each iteration
will sample a (different) random subset of n units
without replacement to include in the experiment.

If n is greater than the number of cross-sectional units in the dataset, the option bootstrap is required
to sample units with replacement.

Note that pc_simulate drops all observations with missing data, and users
should interpret n as the sample size conditional on each unit having
at least 1 observation of non-missing post-treatment data (and 1 observation of
non-missing pre-treatment data,
for DD and ANCOVA models).

p(numlist) governs the proportion of units randomized into treatment. The program rounds p*n to the nearest integer,
and randomly chooses this number of treated units in each iteration.

pre(numlist) governs the number of pre-treatment periods for DD and ANCOVA models. For
ONESHOT and POST models, this option must be either pre(0) or missing.

post(numlist) governs the number of post-treatment periods for POST, DD, and ANCOVA models. For
ONESHOT model, this option must be either post(1) or missing.

The total number of periods, pre + post, cannot exceed the number of time periods in the dataset. Except
in very short panels (i.e., pre<5 or post<5), including additional pre- or post-treatment periods should weakly increase statistical power.

alpha(#) assigns a Type-I error rate (or false discovery rate). The conventional significance level is alpha=0.05, which is also the default value.

onesided toggles a one-sided hypothesis test, instead of the default two-sided test. For
mde>0, the one-sided null hypothesis becomes H_0: b²0. For
mde<0, the one-sided null hypothesis becomes H_0: b³0.

Model options

absorb(varlist) allows users to add fixed effects to the default specifications above.

Note that for ONESHOT, POST, and ANCOVA models, fixed effects are not required for identification, but may improve precision. (This
is especially true for timevar fixed effects in POST and ANCOVA models.)

DD model requires fixed effects, and this option will default to absorb(panelvar timevar) if missing. However,
users may specify alternative fixed effect variables, e.g. absorb(panelvar month year), if timevar=month_year.

controls(varlist) allows users to add linear control variables to default specifications, including linear time trends. Control
variables cannot also be used as fixed effects.

vce(vcetype) may be unadjusted, robust or cluster fvvarlist,
which are the three vcetypes supportd by reghdfe. This
program does not support vce(vcetype, subopt).

In a randomized experiment, unadjusted standard errors will yield correct inference for cross-sectional specifications
(i.e. ONESHOT model, ANCOVA model with post=1)
and for two-period panel specifications with unit fixed effects
(i.e. DD model with pre=1 and post=1).

For all other experiments, unadjusted standard errors will lead to incorrect inference in the presence of within-unit serial correlation
(i.e. models with 3+ time periods, or with 2+ time periods and no unit fixed effects).
Standard errors clustered by unit allow for arbitrary within-unit error correlations, and the program uses vce(cluster panelvar) as a default in these cases.

Unit-level randomization obviates the need for clustering along other dimensions, even in the presence of correlated errors within groups of units,
or correlated errors within time periods across units.

Note that the cluster-robust variance estimator is biased with fewer than ~30 clusters,
and that pc_simulate will not correct for such bias in experimental designs with too few clusters.

Simulation and output options

bootstrap controls whether the program samples cross-sectional units with or without replacement.
The default is to sample without replacement (if bootstrap is not specified), which requires n to be weakly less than the
number of units in the dataset.

If bootstrap is specified, each simulation will draw n cross-sectional units with replacement from the existing dataset,
and n may be larger or smaller than the number of units in the dataset.

For stratified randomization, bootstrap draws units with replacement from within each randomization cell.

For cluster randomization, bootstrap draws whole clusters with replacement. If the option sizecluster is specified,
bootstrap also draws individual units with replacement from within each cluster (yielding clusters of homogeneous size).

(This option replaces the old program pc_bootstrap_units, which is now obsolete.)

nsim(#) governs the number of simulations (or iterations) performed number of simulations performed for each set of experimental design parameters. A
"set" of parameters denotes each combination of mde(numlist), n(numlist),
p(numlist), pre(numlist), and post(numlist).

The program defaults to nsim=500. Users
may begin with fewer simulations (e.g. nsim=100) to narrow down a range of parameter values. However,
nsim of at least 1000 (or even 5000) is recommended for more precise convergence (although this does increase runtime).

outfile(filename) specifies the name of the .csv output file that stores pc_simulate results in the current directory.

If not specified, the program will generate the default filename="c(filename)_power.csv". If
the dataset in memory (i.e. c(filename)) is stored in a temp folder,
the default becomes filename="pc_simulations_power.csv".

Option append will append pc_simulate results below previous results in an existing filename.
Option replace will overwrite an existing filename.
If filename exists, then either append or replace is required.

Design options

collapse estimates a collapsed regression model. For
POST and ANCOVA models, this collapses all post-treatment observations
to a unit-specific average, and estimates a cross-sectional regression.

For DD models, this collapses to separate pre-treatment and post-treatment unit-specific averages, and estimates a two-period pre/post panel. In
a collapsed DD model, the default fixed effects become r(panelvar) and a post-period dummy variable.

Collapsing data eliminates within-unit serial correlation, and the default standard
errors for all collapsed models are vce(unadjusted), which yields correct inference.

To use the option collapse, fixed effects variables cannot vary along both unit and time dimensions.

stratify(varlist) allows users to simulate stratified randomization, where randomization occurs within cells defined by egen cell = group(varlist).
For stratify(gender age_group), each gender-age_group combination would represent a separate randomization cell.
Each panelvar unit must be nested within a single cell.

Importantly, when using this option, n(numlist) governs the number of units in each cell, and p*n units per cell will be assigned to treatment.
(The dataset must either contain at least n units in each randomization cell, or the option bootstrap must be specified.)

Note that pc_simulate still reports power associated with estimating sample-wide average treatment effects, even when simulating a stratified randomization.

Note also that in order to conduct proper inference, users must either include stratification variable(s) as fixed effects via option absorb(varlist),
or cluster by the level(s) of stratification via option vce(vcetype).

idcluster(groupvar) accommodates cluster randomization, where randomization occurs at the groupvar level.
For example, if groups of panelvar units are located in villages, then idcluster(village_id) will randomize whole villages
into treatment (panelvar units must fully nest within groupvar clusters).

Importantly, when using this option, n(numlist) governs the number of clusters
(not the number of units), and p*n cluster will be assigned to treatment.
(The dataset must either contain at least n distinct clusters, or the option bootstrap must be specified.)

Note that pc_simulate still reports power associated with estimating average treatment effects at the unit level.

Note also that in order to conduct proper inference, users should cluster standard errors by groupvar
(i.e. the level of randomization) via option vce(vcetype).

sizecluster(#) is a cluster randomization sub-option that defines the number of panelvar units within each groupvar cluster.
If combined with bootstrap, each simulation will draw units with replacement from within each cluster (yielding clusters of the proscribed size).
If not combined with bootstrap, then all clusters must contain sufficient units to achieve the proscribed size by drawing without replacement.

If sizecluster(#) is not specified, pc_simulate will not alter the size of individual clusters.
The program supports two cases: (i) clusters of homogeneous size, or (ii) potentially heterogeneous cluster sizes as given by the existing dataset.

pcluster(numlist) is a cluster randomization sub-option that defines the intensity of treatment within each groupvar cluster. The program supports 3 cases:

(1): If pcluster is not specified, the default is to treat 100% of units within treated clusters (i.e. every unit within a treated cluster is treated,
and every unit with a control cluster is not treated).

(2): If pcluster has a single value, all treated clusters will have this (uniform) treatment intensity.
For example, pcluster(0.6) yields a 60% treatment intensity,
whereby 60% of units in treated clusters are randomized into treatment. The other 40% of units in treated clusters are not treated,
and every unit with a control cluster is likewise not treated.

(3): If pcluster has multiple values, treated clusters will have heterogeneous treatment intensities,
assigned in equal proportions across treated clusters. For p(0.25) and pcluster(0.4 0.7 1),
25% of clusters receive 0%, 40%, 70%, and 100% treatment intensity (respectively).

Note that as with option p(numlist), pc_simulate rounds to the nearest integer when randomizing
(i) unit-level treatment within clusters and (ii) treatment intensities across clusters. For example, suppose n=10,
p=0.667, sizecluster(10), and pcluster(0.25 1). In this case, 7 clusters will be treated (rounding p*n).
4 treated clusters will be assigned treatment intensity 0.25, and 3 treated clusters will be assigned treatment
intensity 1 (since 2 intensities do not split evenly across 7 treated clusters). Within each of the 4 0.25-intensity clusters,
3 out of 10 units will be treated (rounding pcluster*sizecluster).

Also, note that pc_simulate ignores any potential spillover effects within clusters, which are often a primary motivation for randomizing at the cluster level.

Advanced options

[if] [in] restrict the observations to be included in the power calculation, based on user-provided criteria. If
specified, the program begins by keeping only observations that meet these criteria. After
finishing, the program restores the full original dataset.

tstart(# [#]) restricts the range of time periods over which to begin experiment. For example, tstart(4 7)
will simulate experiments beginning in time periods timevar={4,5,6,7} with equal probability.

If only one number is specified, the option will default to tstart(# max(timevar)).

This option indicates the first pre-treatment period in a DD or ANCOVA model, the first post-treatment
period in a POST model, and the only period in a ONESHOT model.

Unless tstart(# [#]) is specified, each iteration will randomly select a (pre+post)-period window from the full range of timevar in the dataset.
For example, if timevar={1,2,...,100} and pre+post=8, then each iteration will simulate an 8-period experiment beginning
in a time period drawn randomly from timevar={1,2,...,93}.

absorbfactor(fvvarlist) allows users to include factor variables as fixed effects, following the reghdfe syntax. This
option may be combined with the simpler absorb(varlist), and it overrides default fixed effects in DD models.

weight(weight) allows users to specify three types of OLS weights: fweights, aweights and pweights. This
option uses the syntax weight(aw=varname), and it does not support functions of weight variables. See weight for further detail.

reghdfeoptions(options) allows users to pass through additional reghdfe options. Most
of these advanced options will not be compatible with pc_simulate,
and most are unnecessary when estimating a randomized experiment.

This program does not support instrumental variables, two-stage least squares, or other non-linear estimation techniques.

Examples

. pc_simulate z, model(ONESHOT) mde(1) weight(fw=pop) n(50(5)100) vce(robust) nsim(50)

. pc_simulate y if year>2000, model(DD) mde(10 20 50) i(id) t(year) n(100(50)300) bootstrap absorb(id year) nsim(100) out(power_calcs) replace

. pc_simulate income, model(POST) mde(5) i(household) t(month) post(1:20) n(100) boot absorbf(i.village#i.month) nsim(200) stratify(village) append

. pc_simulate consumption, model(ANCOVA) mde(-40) onesided i(indiv) t(week) post(10 20 30) pre(10 20 30) n(50 100 150) a(week) nsim(50) vce(cluster indiv) control(age female) replace

Contact

Louis Preonas

Department of Agricultural and Resource Economics

University of Maryland

Email: lpreonas@umd.edu

This program is part of the ssc package pcpanel.

References

This program grew out of analysis conducted in:

Burlig, Fiona, Louis Preonas, and Matt Woerman (2020). "Panel Data and Experimental Design." Journal of Development Economics 144: 102548.

Online Appendix

Additional references include:

Athey, Susan, and Guido W. Imbens (2016). "The Econometrics of Randomized Experiments." Working Paper.

Bertrand, Marianne, Esther Duflo, and Sendhil Mullainathan (2004). "How Much Should We Trust Differences-in-Differences Estimates?" The Quarterly Journal of Economics 119(1): 249-275.

Bloom, Howard S. (1995). "Minimum Detectable Effects: A Simple Way to Report the Statistical Power of Experimental Designs." Evaluation Review 19(5): 547-556.

Cameron, A. Colin, and Douglas L. Miller (2015). "A Practitioner's Guide to Cluster-Robust Inference." Journal of Human Resources 50(2): 317-372.

Campbell, Cathy (1977). "Properties of Ordinary and Weighted Least Square Estimators of Regression Coefficients for Two-Stage Samples." Proceedings of the Social Statistics Section, American Statistical Association: 800-805.

Duflo, Esther, Rachel Glennerster, and Michael Kremer (2007). "Using Randomization in Development Economics Research: A Toolkit." Chap. 61 in Handbook of Development Economics,
edited by Paul T. Schultz and John A. Strauss, 3895-3962. Volume 4. Oxford, UK: Elsevier.

Frison, L., and S. J. Pocock (1992). "Repeated Measures in Clinical Trials: Analysis Using Mean Summary Statistics and its Implications for Design." Statistics in Medicine 11(13): 1685-1704.

McKenzie, David (2012). "Beyond Baseline and Follow-up: The Case for More T in Experiments." Journal of Development Economics 99(2): 210-221.

Moulton, Brent (1986). "Random Group Effects and the Precision of Regression Estimates." Journal of Econometrics 32(3): 385-397.

Rubin, Donald B. (1974). "Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies." Journal of Educational Psychology 66(5): 688-701.