*! version 2.0.0 13Nov2023

Title

attregtest -- implements the regression-based attrition tests proposed in Ghanem et al. (2023). These attrition tests are outcome-specific and can be applied to field experiments with completely, cluster, or stratified randomization.

This new version of the program allows for the inclusion of additional baseline data on the determinants of (or proxies for) the outcome of interest in the attrition tests. The output of this program is a table with the main regression results and the p-values of the internal validity tests for the outcome in question.

Syntax

attregtest
varlist
[if] [in]
,
treatvar(varname)
respvar(varname)
[options]

Main
stratavar(varname) numerical variable with the strata groups that were used to assign treatment status in the field experiment.

timevar(varname) numerical variable that identifies the different follow-up waves in the field experiment. This option conducts the attrition tests pooling all the follow-up waves.

vce(vcetype) vcetype may be robust, or cluster clustervar.

export(filename.format) name and format of the output table that will be exported.

NOTES:

varlist refers to the outcome of interest measured at baseline. This varlist can include one or multiple variables depending on whether the researcher chooses to include baseline data on the determinants of (or proxies for) the outcome of interest in the attrition tests as well. See Section IV.B in the paper for a discussion on the proper use of covariates in the attrition tests.

treatvar must be a single numerical variable with information on treatment status. This program can be applied to settings with multiple treatment arms as long as
the reference treatment (or control group) takes the value of zero.

respvar is a binary variable that takes the value of one if the outcome of interest is observed at follow-up.

- The command attregtest requires that none of the variables in the dataset is named response.

- The command attregtest requires the most updated version of the package [R] estout. You can
update this package using the following code: ssc install estout, replace.

Description

attregtest implements the mean version of the two attrition tests proposed in Ghanem et al. (2023). The first test focuses on the testable implication of the identifying assumption of internal validity for the respondent subpopulation (IV-R), and the second test is based on the testable implication
of the identifying assumption of internal validity for the study population (IV-P). For more details on these regression-based tests, see Appendix A in the published version of the paper.

These attrition tests can be applied to completely and cluster randomized experiments as well as experiments with stratified randomization. Here is a brief description of the null hypotheses and the results reported in each case:

Completely and Cluster Randomized Experiments:

As detailed in Equation 15 in the paper, if the experiment is completely or cluster randomized, the null hypotheses of the mean IV-R
and IV-P tests for the outcome of interest Y_ are given by:

Y_= a + B_{01}R_ + B_{10}T_ + B_{11}T_R_ + e_

(IV-R) H_o: B_{10}=B_{11}=0.

(IV-P) H_o: B_{01}=B_{10}=B_{11}=0.

where T_ denotes treatment status and R_ refers to the outcome-specific response status at follow-up. In this case, the null hypothesis of the IV-R test consists of the mean equality of baseline outcome for treatment and control respondents as well as treatment and control attritors. In contrast, the null hypothesis of the IV-P test
consists of the mean equality of baseline outcome across all treatment/response subgroups.

Adding Covariates to The Test: If the researcher includes additional baseline data on the determinants of (or proxies for) the outcome of interest in the attrition test, the program conducts the tests proposed in Equation 16 in the paper. These attrition tests are conditions on the joint equality of means across all the specified variables.
See Section IV.B in the paper for a discussion on the proper use of covariates in the attrition tests.

Reported Output: In either case, attregtest uses the command [R] esttab
to produce an output table with the main regression results and the p-values of the internal validity tests. The p-values of the tests are stored in r().

Stratified Randomized Experiments:

If the experiment is stratified, the mean testable restrictions of the IV-R and IV-P assumptions are conditional versions of the testable restrictions for completely or clustered experiments. Thus, the option stratavar(varname) must be specified. As described in Equation 18 in the paper, the attrition tests in these settings are given by:

Y_= \Sum_^[a^ + B_{01}^R_ + B_{10}^T_ + B_{11}^T_R_]1 + e_

(IV-R) H_o: B_{10}^=B_{11}^=0 for all s=1,2,...,S.

(IV-P) H_o: B_{01}^=B_{10}^=B_{11}^=0 for all s=1,2,...,S.

where T_ is treatment status, R_ is the response status at follow-up, and S is the total number of strata used in the randomization. In this case, the null hypothesis of the IV-R test are equality restrictions on the mean baseline outcome for treatment
and control respondents as well as treatment and control attritors within each stratum, while the null hypothesis of
the IV-P test are equality restrictions on the mean baseline outcome across all treatment/response subgroups within each stratum.

Since testing the equality of means within each stratum may result in high-dimensional inference issues when the number of strata is large, the program also reports the test results of the implications of these restrictions. These implications, detailed in Equation 19 in the paper, are equality restrictions on the mean baseline outcome across subgroups in a model with strata fixed effects:

Y_= \Sum_^[a^ +B_{01}^R_]1 + B_{10}T_ + B_{11}T_R_ + e_

(IV-R) H_o: B_{10}=B_{11}=0.

Y_= \Sum_^[a^]1 + B_{01}R_ + B_{10}T_ + B_{11}T_R_ + e_

(IV-P) H_o: B_{01}=B_{10}=B_{11}=0.

Reported Output: Since the appropriate type of attrition tests for stratified experiments (testable restrictions vs implications) depends on each setting, attregtest produces an output table with both sets of p-values. The first two p-values correspond to the test of the mean testable restrictions within each stratum, while the last two p-values correspond to the tests of the implications (i.e., model with strata fixed effects). This table also displays the regression coefficients of the model with strata fixed effects (Equation 19 in the paper), but the fixed effects are omitted for brevity. The p-vals of the testable restrictions are stored as r(pIVR1) and r(pIVP1), while the p-vals of the implications of these restrictions are stored as r(pIVR2) and r(pIVP2).

Adding Covariates to The Test: If the researcher includes additional baseline data on the determinants of (or proxies for) the outcome of interest in the attrition test, the program tests the joint versions of the testable restrictions and implications described above across all the specified variables in varlist.
See Section IV.B in the paper for a discussion on the proper use of covariates in the attrition tests.

Options

Main

stratavar(varname) specifies the variable with the strata groups that were used in the field experiment. Variable must be numerical.
If this option is specified, the program will test the null hypotheses of equality restrictions on the mean baseline
outcome for stratified randomized experiments.

timevar(varname) specifies the variable that identifies the different follow-up waves in the field experiment. Variable must be numerical.
If this option is specified, the program will conduct the attrition tests pooling these follow-up waves.
If the researcher is interested in conducting the test for each follow-up, she should do it separately using the [if] option.
For instance, attregtest z1_outcome if wave==1, treatvar(varname) respvar(varname).

vce(vcetype) specifies the type of standard errors that will be used in the regression-based test. Use (robust) to obtain standard errors
that are robust to heteroskedasticity, and (cluster clustvar) to obtain cluster-robust standard errors; see [R] vce_option.

export(filename.format) attregtest uses the command [R] esttab to produce an output table with the
main results of the internal validity tests. Use this option if you wish to export this table. The string filename.format
specifies the name and the format of the file that will be exported. For instance, testresults.csv or testresults.tex.
The default format is .txt.
See [R] esttab for more details on the supported formats.

Examples

This section illustrates how to use the attregtest command to conduct the attrition tests for completely and stratified randomized experiments using a simple example with simulated data. In this simple example, the baseline outcome
(y_b) is determined by two baseline covariates (w_1 and w_2) and a random error, and this random error is assumed to be independent of response status at follow-up.

To run these simulated examples you should download the command [R] randtreat using the following code: ssc install randtreat.

Generate simulated data:

. clear

. set seed 12345

. set obs 1000

Treatment status:

. randtreat, generate(treat) replace

Baseline covariates that are determinants of the outcome:

. gen w1_b = 0.5*rnormal()

. gen w2_b = 0.7*rnormal()

Baseline outcome:

. gen y_b = 1 + 0.25*w1_b + 0.25*w2_b + rnormal()

Outcome-specific response status at follow-up:

. gen resp_y = (uniform() < .5)

Generate strata variable:

. gen id = _n

. gen random = runiform()

. sort random

. gen sex = _n <= 500

1) Tests of internal validity for completely randomized experiment:

. attregtest y_b, treatvar(treat) respvar(resp_y)

2) Tests of internal validity for completely randomized experiment, including baseline covariates:

. attregtest y_b w1_b w2_b, treatvar(treat) respvar(resp_y)

3) Tests of internal validity for stratified randomized experiment:

. randtreat, generate(treat) replace strata(sex)

. attregtest y_b, treatvar(treat) respvar(resp_y) stratavar(sex)

4) Tests of internal validity for stratified randomized experiment, including baseline covariates:

. randtreat, generate(treat) replace strata(sex)

. attregtest y_b w1_b w2_b, treatvar(treat) respvar(resp_y) stratavar(sex)

References

Ghanem, Dalia, Hirshleifer, Sarojini, Ortiz-Becerra, Karen. 2023. Testing Attrition Bias in Field Experiments. Journal of Human Resources,
http//doi.org/10.3368/jhr.0920-11190R2.

Authors

Dalia Ghanem

University of California, Davis

dghanem@ucdavis.edu

Sarojini Hirshleifer

University of California, Riverside

sarojini.hirshleifer@ucr.edu

Karen Ortiz-Becerra

University of San Diego

kortizbecerra@sandiego.edu