*! version 3.2 17oct2021

Title

pc_dd_analytic -- Analytic power calculations for difference-in-differences model with serial correlation

Syntax

pc_dd_analytic , [n(numlist) mde(numlist) power(numlist) options]

Required (exactly 2 of 3) [+]
* n(numlist) list of sample sizes for experiment
(the number of cross-sectional units included in the randomization); must be positive integers;
e.g. n(50(10)150).

* mde(numlist) list of minimum detectable effect sizes, in levels (not in percents);
e.g. mde(10 20 30).

power(numlist) list of desired statistical power of experiment; must be between 0 and 1;
default is power(0.80).

Experimental design parameters [+]
p(numlist) list of treatment ratios (the proportion of cross-sectional units to receive treatment);
must be between 0 and 1, e.g. p(0.5 0.75); default is p(0.5).

pre(numlist) list of number of pre-treatment periods; must be positive integer(s),
e.g. pre(2(2)10); default is pre(1).

post(numlist) list of number of post-treatment periods; must be positive integer(s), e.g. post(1 2 3);
default is post(1).

alpha(#) desired Type-I error rate (or false discovery rate); must be between 0 and 1; default is alpha(0.05).

onesided assumes one-sided hypothesis test;
if not specified, default is a two-sided hypothesis test.

Residual variance-covariance (estimated) [+]
+ depvar(depvar) dependent variable of experiment, for estimating idiosyncratic
residual variance and covariances; must be numeric.

i(panelvar) cross-sectional unit of randomization; must be numeric; will default to
stored r(panelvar) if not specified.

t(timevar) time period variable; must be numeric; will default to stored r(timevar) if not specified.

if(if) option to restrict observations over which program estimates idiosyncratic
esidual variance and covariances of depvar; e.g. if(year>2000).

in(in) option to restrict observations over which program estimates idiosyncratic
residual variance and covariances of depvar; e.g. in(1/1000).

Residual variance-covariance (assumed) [+]
+ variance(numlist) list of idiosyncratic residual variances; must be positive,
e.g. variance(5(1)10).

+ sd(numlist) list of idiosyncratic residual standard deviations; must be positive,
e.g. sd(1 2 3); a substitute for variance(numlist).

ar1(numlist) list of AR(1) coefficients, to model within-unit serial correlation of idiosyncratic residuals;
must be between -1 and 1, e.g. ar1(0.1(0.1)0.9).

Average covariances (manually input) [+]
avgcov(pre post cross) average within-unit covariances of idiosyncratic residuals,
between pre-treatment periods, between post-treatment periods, and across pre/post-treatment periods, respectively;
equivalent to stored results from pc_dd_covar: avgcov(r(cov_pre) r(cov_post) r(cov_cross));
use with caution, see details below.

avgcor(pre post cross) average within-unit correlations of idiosyncratic residuals,
between pre-treatment periods, between post-treatment periods, and across pre/post-treatment periods, respectively;
equivalent to avgcov(r(cov_pre) r(cov_post) r(cov_cross)) divided by variance(numlist);
use with caution, see details below.

ncovest(#) number of units used to estimate avgcov(pre post cross);
equivalent to stored result r(n_units) from pc_dd_covar;
treats manually input variance (sd) and covariances (correlations) as estiamted parameters.

trueparameters treats manually input variance (sd) and covariances (correlations) as true parameters of the DGP.

Output options [+]
outfile(filename) .txt file that stores results in current directory;
if not specified, output will not be stored but will display in console.

[append|replace] append adds results to an existing .txt file; replace is
required to overwrite existing .txt file (if append is
not specified).

* At least 1 of n(numlist) and mde(numlist) is required. If only 1 is specified,
power(numlist) will default to power=0.8 if missing.

+ Exactly 1 of the following options is required: depvar(depvar), variance(numlist), or sd(numlist).
Options ar1(numlist), avgcov(pre post cross), and avgcor(pre post cross) may not be combined with
depvar(depvar), or with each other.

Description

This program performs analytic power calculations for randomized experiments with difference-in-differences designs. It
accommodates arbitrary serial correlation in the outcome variable, and will also directly estimate the correlation structure
of an existing dataset.

pc_dd_analytic operationalizes the serial-correlation-robust (SCR) DD power calculation formula developed in

Burlig, Fiona, Louis Preonas, and Matt Woerman (2020). "Panel Data and Experimental Design." Journal of Development Economics 144: 102548.

This paper demonstrates that standard power calculation tools ignore non-constant serial correlation present in panel data,
which can lead to dramatically incorrect power calculations. While
power calculations by simulation are recommended, this program is appropriate in cases where
pre-existing data do not exist, or are not reliable. This
program can also help to narrow the range of potential parameters values analytically, before undergoing
more time-consuming simulations.

Users must provide exactly 2 of the following 3 parameters, allowing pc_dd_analytic to solve for the third:

- n(numlist) sample size (number of cross-sectional units in the experiment)

- mde(numlist) minimum detectable effect (in units of the outcome variable, not in percents)

- power(numlist) statistical power (probability of correctly rejecting a false null
given the true effect size mde)

(If power is not specified and one of n or mde is missing, the program will assume the default
power=0.80 and solve of the missing parameter.)

Power calculations assume a standard panel fixed effects/difference-in-differences model, with a treatment
indicator D_it, unit fixed effects fe_i, time period fixed effects fe_t, and serially correlated errors e_it:

- Y_it = b*D_it + fe_i + fe_t + e_it

The simulation-based program pc_simulate accommodates more flexible regression specifications.

Options

Experimental design parameters

p(numlist) denotes the proportion of units in the experiment that are randomized into treatment.

pre(numlist) governs the number of pre-treatment periods in the experiment. Will default to pre=1 if not specified.

post(numlist) governs the number of post-treatment periods in the experiment. Will default to post=1 if not specified.

alpha(#) assigns a Type-I error rate (or false discovery rate). The conventional significance level
is alpha=0.05, which is also the default value.

onesided toggles a one-sided hypothesis test, instead of the default two-sided test. (For analytic power
calculations, the direction of the one-sided test does not matter.)

Estimated vcov options

The strength of pc_dd_analytic is that it parametrizes analytical DD power calculations by
directly estimating the correlation structure of an existing dataset. "Under the hood" of pc_dd_analytic,
the subprogram pc_dd_covar estimates the idiosyncratic residual variance and
covariances of the desired outcome variable (depvar). The resulting power calculations are able to
capture the complex dependencies inherent to real data.

depvar(depvar) denotes the dependent variable of choice, from the dataset in memory.

i(panelvar) and t(timevar) denote the cross-sectional unit of randomization and time period variables, respectively. They
will each be partialed out of depvar as fixed effects.
Both options must be specified together with depvar(depvar), unless r(panelvar) and r(timevar) are already stored in memory.

if(if) and in(in) allow users to specify a subset of depvar observations over
which to estimate residual variance and covariances, if desired.

Please see the pc_dd_covar help documentation for details on how this program estimates
variance and covariance parameters to be fed into the analytical power calculation formula.

Assumed vcov options

In the absence of representative pre-existing data, users can manually input assumed variance and covariance parameters.

variance(numlist) allows users to provide a list of residual variances. Note
that these are not variances of the composite error term, but rather of the idiosyncratic error term,
after partialing out unit and time period fixed effects (i.e., e_it in the equation above).
For users accustomed to the notation of McKenzie (2012) (or sampsi), this variance (var_e) combines the composite variance
(var_Y) and the intracluster correlation (rho): var_e = var_Y * (1-rho).

This option cannot be combined with depvar(depvar) or sd(numlist).

sd(numlist) allows users to provide a list of residual standard deviations, i.e. the square roots of variance(numlist).

This option cannot be combined with depvar(depvar) or variance(numlist).

ar1(numlist) allows users to model serially correlated (idiosyncratic) errors using a simple AR(1) process. This
assumes that each cross-sectional unit's error terms evolve by the process:

e_it = ar1*e_i(t-1) + v_it, where v_it is an i.i.d. white noise term.

While AR(1) represents an imperfect approximation of the complex covariance structures found in real-world panel data,
it is unrealistic to assume i.i.d. error terms in a panel data setting (even in a randomized experiment). In
the absence of a pre-existing dataset, users can enter a range of AR(1) coefficients to test how
weaker/stronger serial correlation changes the relationship between sample size, minimum detectable effect, and statistical power.

ar1(numlist) can only be used in combination with variance(numlist) or sd(numlist). However,
variance(numlist) may also be combined with option avgcov(pre post cross) or avgcor(pre post cross).
See discussion below.

If variance(numlist) or sd(numlist) alone is specified, then the resulting power calculations will assume i.i.d. idiosyncratic errors with no serial correlation.

Average covariances (manually input)

avgcov(pre post cross) allows users to manually input average within-unit covariances of the idiosyncratic errors. These terms are, in order: the
average covariance between pre-treatment periods;
the average covariance between post-treatment periods;
and the average covariance across pre-treatment and post-treatment periods.

These three terms are equivalent to those estimated by pc_dd_covar and stored as avgcov(r(cov_pre) r(cov_post) r(cov_cross)), respectively.

These average covariances depend on the number of pre- and post-treatment periods,
because residuals from time periods farther apart are likely to be less correlated,
thereby reducing the average across all off-diagonal covariances. Hence, if
avgcov is specified, options pre(#) and post(#) can only accommodate one value each. Similarly,
because residual covariances scale with the residual variance, option variance(#) (or sd(#))
can only accommodate one value when combined with avgcov. (Option
depvar(depvar) performs the same operation under the hood, by using pc_dd_covar
to calculate new covariances for each combination of parameters.)

If pre=1 (post=1) then the average residual covariance between pre-treatment
(post-treatment) periods is vacuously defined. However, avgcov(pre post cross) still
requires three inputs, and the program simply ignores any vacuously defined values.

These three terms correspond to psi^B_[omega^hat], psi^A_[omega^hat], and psi^X_[omega^hat] in the
notation of Burlig, Preonas, and Woerman (2020). Please refer to
Appendix D.1
of this paper, and to the pc_dd_covar help documentation for further detail.

avgcor(pre post cross) allows users to manually input three average within-unit correlation terms, rather than covariance terms.
This option is equivalent to avgcov(pre post cross) divided by the idiosyncratic residual variance variance(numlist) (or sd(numlist)^2).
It performs identically to avgcov(pre post cross), with one exception: whereas avgcov(pre post cross) can only accommodate a single
variance (or sd), avgcor(pre post cross) can accommodate multiple inputs for variance(numlist) (or sd(numlist)).

Importantly, avgcor(pre post cross) is not equivalent to the options r0(), r1(), and r01() from the depreciated function sampsi.
Whereas the latter characterized correlations in the composite error term, avgcor(pre post cross) characterizes correlations in the idiosyncratic error
term, after partialing out unit and time fixed effects. Whereas r0=r1=r01 implied constant serial correlation in the outcome variable,
avgcor(pre post cross) with pre=post=cross implies non-constant serial correlation in the outcome variable.

When specifying either avgcor(pre post cross) or avgcor(pre post cross) (which cannot be combined), users must also specify either
ncovest(#) or trueparameters.

ncovest(#) specifies the number of cross-sectional units
used to estimate the four values variance(#) (or sd(#)) and avgcov(pre post cross) (or avgcor(pre post cross)). This
is used in the correction factors that allow pc_dd_analytic to estimate unbiased
power calculations using (biased) estimates of these variance-covariance parameters. (See Appendix
E from Burlig, Preonas, and Woerman (2020) for more detail.)

This option tells pc_dd_analytic to treat variance() (or sd()) and avgcov() (or avgcor()) as estimated parameters.
In other words, the program will apply estimation correction factors to the variance/covariance terms in the analytic power calculation formula.
This will yield inaccurate power calculations in cases where users estimate the variance from residuals, but do not also estimate covariances
from the same residuals.

Estimating variance and covariance parameters from regression
residuals will not replicate the true parameters governing the DGP of the error term.
To take one simple example,
suppose e_it = 0.8*e_i + v_it, where var(v_it)=3.6, implying that var(e_it)=10.
In a model with pre=1 and post=1, the average
covariance across pre- and post-treatment periods is simply the AR(1) correlation scaled by the variance
of e_it, or cov(e_ipre,e_ipost)=8. However,
partialing out fixed effects from the outcome variable recovers residuals that are not equal to e_it,
due to estimation error. The residual
variance will be less than 10, and the average cross-period residual covariance will be negative.

trueparameters tells pc_dd_analytic to treat variance() (or sd()) and avgcov() (or avgcor()) as true parameters.
In other words, the program will assume the variance/covariance terms in the analytic power calculation formula characterize the true DGP of data.
This will yield inaccurate power calculations in cases where users have estimated either the variance or covariance terms from residuals,
because of the estimation error inherent in doing so.

Toggling this option only makes sense in cases where users are "guessing" the variance/covariance structure. Because avgcov(pre post cross)
and avgcor(pre post cross) are functions of both the data structure and the panel length, we recommend that users "guess"
either using option ar1(numlist) or using option depvar(depvar) on simulated data.

Output options

outfile(filename) specifies the name of the .txt output file that stores pc_dd_analytic results in the current directory.
If not specified, the program will not store its results, but they will still display in the Stata console.

Option append will append pc_dd_analytic results below previous results in an existing filename.
Option replace will overwrite an existing filename.
If filename exists, then either append or replace is required.

Examples

. pc_dd_analytic, n(80(5)160) pre(3) post(3) var(1) ar1(0.9)

. pc_dd_analytic, n(100 200 300) power(0.90) pre(5) post(20) var(63.2) ar1(0.1(0.1)0.9) alpha(0.01) onesided

. pc_dd_analytic, mde(20) power(0.8) pre(1(1)10) post(1(1)10) p(0.5) depv(y) i(person_id) t(month) out(power_calcs.txt) replace

. pc_dd_analytic, n(50 60 70) mde(20 15 10) depv(income) i(village_id) t(year) if(village_id!=999) out(pc_test.txt) append

Contact

Louis Preonas

Department of Agricultural and Resource Economics

University of Maryland

Email: lpreonas@umd.edu

This program is part of the ssc package pcpanel.

References

This program implements the analytic power calculation formula derived in:

Burlig, Fiona, Louis Preonas, and Matt Woerman (2020). "Panel Data and Experimental Design." Journal of Development Economics 144: 102548.

Online Appendix

Additional references include:

Athey, Susan, and Guido W. Imbens (2016). "The Econometrics of Randomized Experiments." Working Paper.

Bertrand, Marianne, Esther Duflo, and Sendhil Mullainathan (2004). "How Much Should We Trust Differences-in-Differences Estimates?" The Quarterly Journal of Economics 119(1): 249-275.

Bloom, Howard S. (1995). "Minimum Detectable Effects: A Simple Way to Report the Statistical Power of Experimental Designs." Evaluation Review 19(5): 547-556.

Cameron, A. Colin, and Douglas L. Miller (2015). "A Practitioner's Guide to Cluster-Robust Inference." Journal of Human Resources 50(2): 317-372.

Campbell, Cathy (1977). "Properties of Ordinary and Weighted Least Square Estimators of Regression Coefficients for Two-Stage Samples."
Proceedings of the Social Statistics Section, American Statistical Association: 800-805.

Duflo, Esther, Rachel Glennerster, and Michael Kremer (2007). "Using Randomization in Development Economics Research: A Toolkit."
Chap. 61 in Handbook of Development Economics,
edited by Paul T. Schultz and John A. Strauss, 3895-3962. Volume 4. Oxford, UK: Elsevier.

Frison, L., and S. J. Pocock (1992). "Repeated Measures in Clinical Trials: Analysis Using Mean Summary Statistics and its Implications for Design." Statistics in Medicine 11(13): 1685-1704.

McKenzie, David (2012). "Beyond Baseline and Follow-up: The Case for More T in Experiments." Journal of Development Economics 99(2): 210-221.

Moulton, Brent (1986). "Random Group Effects and the Precision of Regression Estimates." Journal of Econometrics 32(3): 385-397.

Rubin, Donald B. (1974). "Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies." Journal of Educational Psychology 66(5): 688-701.