{smcl} {* *! version 1.1 03jul2020}{...} {title:Title} {pstd}{hi:powermap} {hline 2} Power heat maps for experimental design using multiple periods{p_end} {title:Syntax} {p 8 15 2} {cmd:powermap} [{cmd:,} {it:options}]{p_end} {synoptset 23 tabbed} {synopthdr} {synoptline} {syntab:Main} {p2coldent:* {opt n(#)}}study sample size; must be a positive integer ({help powermap##options:see definition below}){p_end} {p2coldent:* {opt pow:er(#)}}desired statistical power of the experiment (1-type II error); must be between 0 and 1{p_end} {synopt:{opt mde(#)}}minimum detectable effect{p_end} {synopt:{opt me:thod(method)}}estimation method, where {it:method} is: {opt post}, {opt change}, {opt ancova}{p_end} {synopt:{opt rho(#)}}serial correlation of the outcome variable{p_end} {synopt:{opt sd(#)}}standard deviation of the outcome variable{p_end} {syntab:Optional} {synopt:{opt ro:unds(#)}}total pre/post treatment rounds; default is {cmd:{ul:ro}unds(20)}{p_end} {synopt:{opt pt:reat(#)}}proportion of the study sample allocated to the treatment group; must be between 0 and 1; default is {cmd:{ul:pt}reat(0.5)}{p_end} {synopt:{opt alp:ha(#)}}significance level (type I error); must be between 0 and 1; default is {cmd:{ul:alp}ha(0.05)}{p_end} {synopt:{opt one:sided}}one-sided hypothesis test; default is a {opt two-sided test}{p_end} {synopt:{opth saveg:raph(filename)}}save graph in the current path/folder{p_end} {synopt:{opth saved:ata(filename)}}store inputs and outputs of the power/sample calculations into a .dta file in the current path/folder{p_end} {synoptline} {p2colreset}{...} {p 4 6 2}* Only 1 of {opt n(#)} and {opt pow:er(#)} is required. If {opt n(#)} is specified, {cmd:powermap} plots the statistical power of all the pre-post rounds combinations for the given study sample size {opt n(#)}. If {opt pow:er(#)} is specified, the program plots the study sample size required by each pre-post rounds combination to achieve a power of {opt power(#)}.{p_end} {title:Description} {pstd} {cmd:powermap} creates power/sample heat maps for experimental design with multiple periods. {cmd:powermap} conducts analytical power/sample calculations and plots a heat map for all the combinations of pre-post treatment periods, hence illustrating the trade-offs of multiple measurements and the size of cross-sectional samples for experimental design. The program allows two approaches: (i) a heat map of the statistical power of all the pre-post rounds combinations for a given study sample size {opt n(#)}; (ii) a heat map of the total study sample size required by each pre-post rounds combination to achieve a desired power of {opt power(#)}. See some {help powermap##examples:examples} below.{p_end} {title:Dependencies} {pstd} {cmd:powermap} uses the {help sampsi:sampsi} command to conduct analytical power/sample calculations. Even though {help sampsi:sampsi} is no longer an officially supported Stata package, it allows to estimate power/sample under multiple periods, relative to the more recent {help power:power} command (see {browse "https://dimewiki.worldbank.org/wiki/Power_Calculations_in_Stata#sampsi":DIME's guide to conduct power calculations in Stata}).{p_end} {pstd} {cmd:powermap} also requires {cmd:heatplot} (Jann, 2019), which in turn requires {cmd:palettes} (Jann, 2018), and {cmd:colrspace} (Jann 2019a). To install these packages, type:{p_end} . {stata ssc install heatplot, replace} . {stata ssc install palettes, replace} . {stata ssc install colrspace, replace}{txt} {marker options}{...} {title:Options} {dlgtab:Main} {phang} {opt n(#)} is the total study sample size (cannot be combined with {opt pow:er(#)}). This is, the total number of surveys/data points to be conducted throughout the study. It should not be confused with the {ul:cross-sectional sample size} (i.e. the total number of observations that can be measured in each period/cross-section). Using this fixed {opt n(#)} illustrates the trade-offs between fewer measurements with larger cross-sectional samples vs. more measurements with smaller cross-sectional samples. If {opt n(#)} is specified, {cmd:powermap} plots the statistical power of all the pre-post rounds combinations for the given study sample size {opt n(#)}. Must be a positive integer.{p_end} {phang} {opt pow:er(#)} is the desired power of the experimental design and equivalent to (1-type II error) (cannot be combined with {opt n(#)}). If {opt pow:er(#)} is specified, {cmd:powermap} plots the study sample size required by each pre-post rounds combination to achieve a power of {opt power(#)}. Must be between 0 and 1 and, if not specified, the default is {cmd:{ul:pow}er(0.8)}.{p_end} {phang} {opt mde(#)} is the minimum detectable effect. This is, the expected treatment effect assuming a zero mean for the control group. {opt mde(#)} can also be negative, in case the treatment is expected to have a negative effect.{p_end} {phang} {opt me:thod(post change ancova)} is the approach to estimate the treatment effects. The {it:post} option ignores the pre-treatment periods and allocates all the study sample size to the post-treatment periods (i.e. a design with 1 pre and 1 post periods will produce the same heat map as a design with 10 pre and 1 post periods, as the {it:post} option only considers the latter).{p_end} {phang} {opt ro:unds(#)} is the total pre/post treatment periods. This is, the extent of the x-axis and y-axis of the heat map including all the combinations of pre/post rounds. (e.g. {cmd:{ul:ro}unds(10)} will produce a 10x10 grid with the power/sample calculations for each combination between {it:pre} = 1,2,3,...,10 and {it:post} = 1,2,3,...,10. If not specified, the default is {cmd:{ul:ro}unds(20)}.{p_end} {phang} {opt rho(#)} is the serial correlation of the outcome variable. {opt powermap} assumes constant serial correlation rather than arbitrary non-i.i.d. or more complex autocorrelation structures. See, for instance, {browse "https://www.sciencedirect.com/science/article/abs/pii/S030438781200003X":McKenzie (2012)}; {browse "https://www.sciencedirect.com/science/article/pii/S030438782030033X":Burlig et. al (2020)}.{p_end} {phang} {opt sd(#)} is the standard deviation of the outcome variable.{p_end} {dlgtab:Optional} {phang} {opt pt:reat(#)} is the proportion of the study sample allocated to the treatment group; must be between 0 and 1; default is {cmd:{ul:pt}reat(0.5)}.{p_end} {phang} {opt alp:ha(#)} is the significance level (type I error); must be between 0 and 1; default is {cmd:{ul:alp}ha(0.05)}.{p_end} {phang} {opt one:sided} assumes a one-sided hypothesis test; default is a {opt two-sided hypothesis test}.{p_end} {phang} {opth saveg:raph(filename)} saves the graph in the current path/folder. If not specified, the data is not stored.{p_end} {phang} {opth saved:ata(filename)} stores the input and output of the power/sample calculations into a .dta file. If not specified, the data is not stored.{p_end} {marker examples}{...} {title:Examples} {dlgtab:Power heat maps} {pstd} Power for an experiment with a budget of 2000 surveys; an MDE and s.d. of 10 and 100, respectively; ρ = 0.5; using an ANCOVA estimation:{p_end} . {stata powermap, n(2000) mde(10) sd(100) rho(0.5) method(ancova)} {pstd} Power for different serial correlation levels:{p_end} . {stata powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova)} . {stata powermap, n(2000) mde(10) sd(100) rho(0.9) method(ancova)} {pstd} Power for different estimation approaches:{p_end} . {stata powermap, n(2000) mde(10) sd(100) rho(0.1) method(post)} . {stata powermap, n(2000) mde(10) sd(100) rho(0.1) method(change)} {pstd} Power for shorter-longer panels:{p_end} . {stata powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova) rounds(10)} . {stata powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova) rounds(30)} {pstd} Power assuming (i) uneven sample allocation to the treatment group, (ii) one-sided hypothesis test:{p_end} . {stata powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova) ptreat(.75)} . {stata powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova) onesided} {pstd} Storing power heat maps and data:{p_end} . {stata powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova) savegraph(powermaps_graph)} . {stata powermap, n(2000) mde(10) sd(100) rho(0.1) method(ancova) savedata(powermaps_data)} {dlgtab:Sample heat maps} {pstd} Study sample for an experiment with 80% power; an MDE and s.d. of 5 and 20, respectively; ρ = 0.5; using an ANCOVA estimation:{p_end} . {stata powermap, power(0.8) mde(5) sd(20) rho(0.5) method(ancova)} {pstd} Study sample for different serial correlation levels:{p_end} . {stata powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova)} . {stata powermap, power(0.8) mde(5) sd(20) rho(0.9) method(ancova)} . {stata powermap, power(0.8) mde(5) sd(20) rho(0.5) method(post)} . {stata powermap, power(0.8) mde(5) sd(20) rho(0.9) method(post)} {pstd} Study sample for different estimation approaches:{p_end} . {stata powermap, power(0.8) mde(5) sd(20) rho(0.1) method(post)} . {stata powermap, power(0.8) mde(5) sd(20) rho(0.1) method(change)} {pstd} Study sample for shorter-longer panels:{p_end} . {stata powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova) rounds(10)} . {stata powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova) rounds(30)} {pstd} Study sample assuming (i) uneven sample allocation to the treatment group, (ii) one-sided hypothesis test:{p_end} . {stata powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova) ptreat(.25)} . {stata powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova) onesided} {pstd} Storing sample heat maps and data:{p_end} . {stata powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova) savegraph(samplemaps_graph)} . {stata powermap, power(0.8) mde(5) sd(20) rho(0.1) method(ancova) savedata(samplemaps_data)} {title:Stored results} {synoptset 15 tabbed}{...} {p2col 5 15 19 2: Scalars}{p_end} {p2coldent:* {cmd:r(N)}}study sample size{p_end} {p2coldent:* {cmd:r(power)}}statistical power{p_end} {synopt:{cmd:r(mde)}}minimum detectable effect{p_end} {synopt:{cmd:r(rho)}}serial correlation of the outcome variable{p_end} {synopt:{cmd:ptreat(#)}}proportion of the study sample allocated to the treatment group{p_end} {synopt:{cmd:alpha(#)}}significance level (type I error){p_end} {p2col 5 15 19 2: Macros}{p_end} {synopt:{cmd:r(method)}}estimation method{p_end} {synopt:{cmd:r(test)}}one-sided/two-sided hypothesis test{p_end} {p 4 6 2}* Only 1 of the two scalars is returned, according to which of the {opt n(#)} or {opt pow:er(#)} options is specified.{p_end} {hline} {title:Contact} {pstd} Cristhian Pulido{break} Department of Economics{break} University of Sussex{break} E-mail: {browse "mailto:c.pulido@sussex.ac.uk":c.pulido@sussex.ac.uk}{p_end} {hline} {title:References} {phang} Burlig, F., Preonas, L., & Woerman, M. (2020). Panel data and experimental design. Journal of Development Economics, 102458.{p_end} {phang} Jann, B. (2019b). Heat (and hexagon) plots in Stata. Presentation at London Stata Conference 2019. Available from {browse "http://ideas.repec.org/p/boc/usug19/24.html"}.{p_end} {phang} McKenzie, D. (2012). Beyond baseline and follow-up: The case for more T in experiments. Journal of development Economics, 99(2), 210-221.{p_end} {phang} Pulido, C. (2020). How efficient are panel RCTs? The case for simulation-based methods for experimental design. (Forthcoming).{p_end}