*! version 1.0.1 10oct2019

Title

retrodesign -- Assessing Type-S (Sign) and Type-M (Magnitude) Errors

Syntax

retrodesign
effect-size
,
se(#)
[
alpha(#)
df(#)
seed(#)
reps(#)
]

effect-size(s) can be entered as a single numeric value or as a numlist

se(#) standard error of the estimate; se() is required

alpha(#) set the alpha level; default is 0.05

df(#) degrees of freedom; When df() is specified, the Gelman and Carlin (2014) simulation method is used for computing type-M error

seed(#) set random-number seed to # when computing type-M error; this option only works when df() is specified

reps(#) set number of simulated observations when computing type-M error. This option only works when df() is specified; default is 10,000

Description

retrodesign computes power, type-S, and type-M errors for one or more specified effect sizes. A type-S (sign) error indicates the probability of an effect size
estimate being in the wrong direction, and a type-M (magnitude) error indicates the factor by which the magnitude of an effect might be overestimated -- given that the
test statistic is statistically significant (Gelman and Carlin 2014).

Gelman and Carlin (2014) propose computing the type-M error using the Student's t distribution. This method is implemented in retrodesign when the
user specifies df() (and optionally, reps() and seed()). Lu, Qiu, and Deng (2019) propose a closed form solution for computing the type-M error.
This method is implemented in retrodesign when the df() is not specified. retrodesign produces identical results to those computed in the retrodesign
package for R (https://cran.r-project.org/web/packages/retrodesign/index.html).

Options

se(#) specifies the standard error of the estimate; se() is required.

alpha(#) specifies the desired alpha level; default is 0.05.

df(#) specifies the degrees of freedom used to compute the type-M error when implementing the method by Gelman and Carlin (2014); df() cannot exceed 9.007e+15.

seed(#) sets the random-number seed. Specifying this option ensures reproducibility of the computed type-M error when implementing the method by Gelman and Carlin (2014).

reps(#) sets the number of simulated observations to generate when computing the Type M error when implementing the method by Gelman and Carlin (2014); default is 10,000.

Examples

"Beauty and sex ratios" example from Gelman and Carlin (2014). We start by using getregstats (if installed) to compute the standard error based on given point estimate of 8.0 and p-value of 0.015.

. getregstats 8.0, model(lin) pval(0.015)

Next, we use retrodesign with a single true estimate of 0.1, and the std err of 3.3. We compute
the type-M error using the Lu, Qiu, and Deng (2019) method.

. retrodesign 0.1, se(3.3) alpha(0.05)

Same as above, but a range of true effect sizes is applied.

. retrodesign 0.1 0.3 1 2 3, se(3.3) alpha(0.05)

Same as above, but the Gelman and Carlin (2014) method is applied by setting df(). Here we set df() to the maximum value allowed and set the seed to allow for reproducible results.

. retrodesign 0.1 0.3 1 2 3, se(3.3) alpha(0.05) df(9007199254740990) seed(1234)

Stored results

retrodesign stores the following in r():

Matrices

r(table) results table

References

Gelman, A. and J, Carlin. 2014. Beyond power calculations: Assessing type S (sign) and type M (magnitude) errors. Perspectives on Psychological Science 9(6):641-651.

Lu, J., Qiu, Y. and A, Deng. 2019. A note on Type S/M errors in hypothesis testing. British Journal of Mathematical and Statistical Psychology 72(1):1-17.

Citation of retrodesign

retrodesign is not an official Stata command. It is a free contribution
to the research community, like a paper. Please cite it as such:

Linden A. (2019). RETRODESIGN: Stata module for computing type-S (Sign) and type-M (Magnitude) errors.
Statistical Software Components, Boston College Department of Economics. http://ideas.repec.org/c/boc/bocode/s458631.html

Authors

Ariel Linden

President, Linden Consulting Group, LLC

alinden@lindenconsulting.org

Also see

Online: power, getregstats (if installed)