*! version 1.1 03jul2020

Title

powermap -- Power heat maps for experimental design using multiple periods

Syntax

powermap
[, options]

Main
* n(#) study sample size; must be a positive integer (see definition below)

* power(#) desired statistical power of the experiment (1-type II error); must be between 0 and 1

mde(#) minimum detectable effect

method(method) estimation method, where method is: post, change, ancova

rho(#) serial correlation of the outcome variable

sd(#) standard deviation of the outcome variable

Optional
rounds(#) total pre/post treatment rounds; default is rounds(20)

ptreat(#) proportion of the study sample allocated to the treatment group; must be between 0 and 1; default is ptreat(0.5)

alpha(#) significance level (type I error); must be between 0 and 1; default is alpha(0.05)

onesided one-sided hypothesis test; default is a two-sided test

savegraph(filename) save graph in the current path/folder

savedata(filename) store inputs and outputs of the power/sample calculations into a .dta file in the current path/folder

* Only 1 of n(#) and power(#) is required. If n(#)
is specified, powermap plots the statistical power of all the pre-post
rounds combinations for the given study sample size n(#). If power(#) is
specified, the program plots the study sample size required by each pre-post
rounds combination to achieve a power of power(#).

Description

powermap creates power/sample heat maps for experimental design with
multiple periods. powermap conducts analytical power/sample calculations
and plots a heat map for all the combinations of pre-post treatment periods,
hence illustrating the trade-offs of multiple measurements and the size of
cross-sectional samples for experimental design. The program allows two
approaches: (i) a heat map of the statistical power of all the pre-post rounds
combinations for a given study sample size n(#); (ii) a heat map of the
total study sample size required by each pre-post rounds combination to achieve
a desired power of power(#). See some examples below.

Dependencies

powermap uses the sampsi command to conduct analytical power/sample
calculations. Even though sampsi is no longer an officially
supported Stata package, it allows to estimate power/sample under multiple
periods, relative to the more recent power command (see DIME's guide to conduct power calculations in Stata).

powermap also requires heatplot (Jann, 2019), which in turn
requires palettes (Jann, 2018), and colrspace (Jann 2019a). To install
these packages, type:

. ssc install heatplot, replace
. ssc install palettes, replace
. ssc install colrspace, replace

Options

Main

n(#) is the total study sample size (cannot be combined with power(#)). This
is, the total number of surveys/data points to be conducted throughout the study. It
should not be confused with the cross-sectional sample size (i.e. the total number
of observations that can be measured in each period/cross-section). Using this fixed n(#) illustrates
the trade-offs between fewer measurements with larger cross-sectional samples vs. more
measurements with smaller cross-sectional samples. If n(#) is specified, powermap plots
the statistical power of all the pre-post rounds combinations for the given study
sample size n(#). Must be a positive integer.

power(#) is the desired power of the experimental design and equivalent
to (1-type II error) (cannot be combined with n(#)). If power(#) is
specified, powermap plots the study sample size required by each pre-post
rounds combination to achieve a power of power(#). Must be between 0 and 1
and, if not specified, the default is power(0.8).

mde(#) is the minimum detectable effect. This is, the expected treatment
effect assuming a zero mean for the control group. mde(#) can also be
negative, in case the treatment is expected to have a negative effect.

method(post change ancova) is the approach to estimate the treatment
effects. The post option ignores the pre-treatment periods and allocates
all the study sample size to the post-treatment periods (i.e. a design with 1
pre and 1 post periods will produce the same heat map as a design with 10 pre
and 1 post periods, as the post option only considers the latter).

rounds(#) is the total pre/post treatment periods. This is, the extent of
the x-axis and y-axis of the heat map including all the combinations of pre/post
rounds. (e.g. rounds(10) will produce a 10x10 grid with the power/sample
calculations for each combination between pre = 1,2,3,...,10 and post
= 1,2,3,...,10. If not specified, the default is rounds(20).

rho(#) is the serial correlation of the outcome variable. powermap assumes
constant serial correlation rather than arbitrary non-i.i.d. or more complex
autocorrelation structures. See, for instance, McKenzie (2012); Burlig et. al (2020).

sd(#) is the standard deviation of the outcome variable.

Optional

ptreat(#) is the proportion of the study sample allocated to the treatment
group; must be between 0 and 1; default is ptreat(0.5).

alpha(#) is the significance level (type I error); must be between 0 and 1; default
is alpha(0.05).

onesided assumes a one-sided hypothesis test; default is a two-sided hypothesis test.

savegraph(filename) saves the graph in the current path/folder. If not
specified, the data is not stored.

savedata(filename) stores the input and output of the power/sample calculations
into a .dta file. If not specified, the data is not stored.

Examples

Power heat maps

Power for an experiment with a budget of 2000 surveys; an MDE and s.d. of 10 and 100, respectively; ρ = 0.5; using an ANCOVA estimation:

. powermap, n(2000) mde(10) sd(100) rho(0.5) method(ancova)

Power for different serial correlation levels:

. powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova)
. powermap, n(2000) mde(10) sd(100) rho(0.9) method(ancova)

Power for different estimation approaches:

. powermap, n(2000) mde(10) sd(100) rho(0.1) method(post)
. powermap, n(2000) mde(10) sd(100) rho(0.1) method(change)

Power for shorter-longer panels:

. powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova) rounds(10)
. powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova) rounds(30)

Power assuming (i) uneven sample allocation to the treatment group, (ii) one-sided hypothesis test:

. powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova) ptreat(.75)
. powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova) onesided

Storing power heat maps and data:

. powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova) savegraph(powermaps_graph)
. powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova) savedata(powermaps_data)

Sample heat maps

Study sample for an experiment with 80% power; an MDE and s.d. of 5 and 20, respectively; ρ = 0.5; using an ANCOVA estimation:

. powermap, power(0.8) mde(5) sd(20) rho(0.5) method(ancova)

Study sample for different serial correlation levels:

. powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova)
. powermap, power(0.8) mde(5) sd(20) rho(0.9) method(ancova)

. powermap, power(0.8) mde(5) sd(20) rho(0.5) method(post)
. powermap, power(0.8) mde(5) sd(20) rho(0.9) method(post)

Study sample for different estimation approaches:

. powermap, power(0.8) mde(5) sd(20) rho(0.1) method(post)
. powermap, power(0.8) mde(5) sd(20) rho(0.1) method(change)

Study sample for shorter-longer panels:

. powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova) rounds(10)
. powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova) rounds(30)

Study sample assuming (i) uneven sample allocation to the treatment group, (ii) one-sided hypothesis test:

. powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova) ptreat(.25)
. powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova) onesided

Storing sample heat maps and data:

. powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova) savegraph(samplemaps_graph)
. powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova) savedata(samplemaps_data)

Stored results

Scalars

* r(N) study sample size

* r(power) statistical power

r(mde) minimum detectable effect

r(rho) serial correlation of the outcome variable

ptreat(#) proportion of the study sample allocated to the treatment group

alpha(#) significance level (type I error)

Macros

r(method) estimation method

r(test) one-sided/two-sided hypothesis test

* Only 1 of the two scalars is returned, according to which of
the n(#) or power(#) options is specified.

Contact

Cristhian Pulido

Department of Economics

University of Sussex

E-mail: c.pulido@sussex.ac.uk

References

Burlig, F., Preonas, L., & Woerman, M. (2020). Panel data and experimental design. Journal of Development Economics, 102458.

Jann, B. (2019b). Heat (and hexagon) plots in Stata. Presentation at London Stata Conference 2019. Available from http://ideas.repec.org/p/boc/usug19/24.html.

McKenzie, D. (2012). Beyond baseline and follow-up: The case for more T in experiments. Journal of development Economics, 99(2), 210-221.

Pulido, C. (2020). How efficient are panel RCTs? The case for simulation-based methods for experimental design. (Forthcoming).