help for rwolf2

Title

rwolf2 -- A more flexible syntax to calculate Romano-Wolf stepdown p-values for multiple hypothesis testing

Syntax

rwolf2
(method1
1
1 [if] [in] [weight] [, options1])

(method2
2
2 [if] [in] [weight] [, options2])

...

(methodN
N
N [if] [in] [weight] [, optionsN]),

indepvars(vars1, vars2, ..., varsN) [options]

Options
indepvars(varlist) This is a required option. Each indepdendent variable should be indicated for which the multiple hypothesis correction is desired.
These refer to variables indicated in each of the independent variable varlists of models specified in the syntax. Variables in each model should be separated by commas.
If a correction is desired for a single independent variable from each model this should be specified as (var1, var2, ..., varN).
It is possible to indicate various independent (treatment) variables which should be corrected across models and this quantity can vary for each model, for example
(var1a var1b, var2, ..., varNa VarNb varNc).

reps(#) Perform # bootstrap replication; default is reps(100).
Where possible prefer a considerably larger number of replications for more precise p-values.

seed(#) Sets seed to indicate the initial value for the pseudo-random number generator. # can be any integer between 0 and 2^31-1.

holm Along with standard output, additionally provide p-values corresponding to the
Holm multiple hypothesis correction.

graph Requests that a graph be produced showing the Romano-Wolf null distribution
corresponding to each hypothesis test considered.

varlabels Name panels on the graph of null distributions using their variable labels
rather than their variable names.

onesided(string) Indicates that p-values based on one-sided tests should be calculated.
Unless specified, p-values based on two-sided tests are provided, corresponding to the null that
each parameter is equal to 0 (or the values indicated in nulls()). In onesided(string),
string must be either "positive", in which case the null is that each parameter is greater
than or equal to 0, or "negative" in which case the null is that each parameter is less than or equal to 0.

nodots Suppress replication dots in bootstrap resamples.

noplusone Calculate the Resampled and Romano-Wolf adjusted p-values without
adding one to the numerator and denominator.

nulls(numlist) Indicates the parameter values of interest used in
each test. If specified, a single scalar value should be indicated for each of the multiple
hypotheses tested, and these should be listed in the same order that variables are
listed as indepvars in the command syntax. If this option is not used, it is assumed
that each null hypothesis is that the parameter is equal to 0.

strata(varlist) Specifies the variables identifying strata. If strata() is specified, bootstrap samples are selected within each stratum when forming the resampled null distributions.

cluster(varlist) Specifies the variables identifying resampling clusters.
If cluster() is specified, the sample drawn when forming the resampled null
distributions is a bootstrap sample of clusters. This option does not cluster standard errors
in each original regression. If desired, this should be additionally specified as an option
in the individual regression models.

idcluster(newvar) In cases where clustered resampling is used, the idcluster
option should be used as used in bsample to specify a new name for resampled clusters, such
that clustering within each bootstrap is based on newvar rather than the original (repeated)
cluster variable. If this option is used, any instances of cluster(var) or vce(cluster var)
will be replaced with cluster(newvar) or vce(cluster newvar) within each bootstrap resample.

verbose Requests additional output, including display of the initial
(uncorrected) models estimated.

usevalid In cases of bootstrap replicates where invalid Studentized test
statistics are generated, these will be removed from the multiple hypothesis correction
procedure. In cases where such
invalid statistics are generated (for example due to no valid underlying estimate or standard error),
a warning will be issued by rwolf2 that the usevalid option should be specified. This is highly
recommended. In cases where bootstrapped models result in errors causing the command to exit with error,
the usevalid option can be used to skip these replicates, and avoid errors with the program.

Description

rwolf2 calculates Romano and Wolf's (2005a,b) step-down adjusted p-values robust to
multiple hypothesis testing. It provides a more general syntax than that provided in
the rwolf command, although the underlying algorithm is the same.
This program follows the resampling algorithm described in
Romano and Wolf (2016), and provides a p-value corresponding to the significance of a
hypothesis test where S tests have been implemented, providing strong control of the
familywise error rate (the probability of committing any Type I error among all
of the S hypotheses tested). The rwolf2 algorithm constructs a null
distribution for each of the S hypothesis tests based on Studentized bootstrap replications
of a subset of the tested variables. Full details of the procedure are described in
Romano and Wolf (2016), additional discussion related to the procedure in Stata is provided
in Clarke, Romano and Wolf (2019).

This command follows a syntax similar to sureg where multiple models of interest
should each be specified in a separate set of parentheses. These models will be estimated,
and bootstrap replicates will be generated, and based on these estimates and bootstrap
replicates, the Romano-Wolf correction will be implemented. The variables of interest
(for which the multiple hypothesis correction is desired) should be indicated in the
required option indepvars(). This syntax can be used to implement any
Stata estimation command that returns a parameter vector and variance-covariance
matrix, as well as other non-native commands including user-written ados such as reghdfe,
ivreg2 (and related programs) and rdrobust.

In each model, optionally, regression weights,
if
or in can be specified. By default, 100 bootstrap replications are run
for each of the S multiple hypotheses. Where possible, a much larger number of replications
should be preferred given that p-values are computed by comparing estimates to a
bootstrapped null distribution constructed from these replications. The number of
replications is set using the reps(#) option, and to replicate
results, the seed(#) should be set.

By default, the re-sampled null distributions are formed using a simple bootstrap
procedure. However, more complex stratified and/or clustered resampling procedures
can be specified using the strata() and cluster() options. The
cluster() option refers only to the resampling procedure, and
not to the standard errors estimated in each original regression model. If the standard
variance estimator is not desired for regression models, this should be indicated
in the syntax of each method indicated in the command syntax.

The command returns the Romano Wolf p-value corresponding to each variable, standard
(bootstrapped) uncorrected p-values, and for reference, the original uncorrected
(analytical) p-value from the initial tests when rwolf2 estimates baseline
regression models. rwolf2 is an e-class command, and a matrix is returned as
e(RW) providing the full set of Romano-Wolf corrected p-values (and other p-values
mentioned above).

Examples

Use the auto dataset to run multiple regressions of various independent variables on a single dependent variable of interest (weight) controlling for trunk and mpg and correcting for multiple testing on weight.

. sysuse auto

. rwolf2 (regress headroom weight trunk mpg) (regress turn weight trunk mpg) (regress price weight trunk mpg) (regress rep78 weight trunk mpg), indepvars(weight, weight, weight, weight) reps(250)

Estimate an IV and OLS model, correcting for a single variable in the IV regression, and two variables in the OLS regression.

. rwolf2 (ivregress 2sls price (weight = length)) (regress price weight trunk), indepvars(weight, weight trunk) verbose

Run multiple hypothesis tests using the National Longitudinal (panel) Survey with an xtreg, fe model plus a standard OLS regression, stratifying resamples by occupation.

Setup

. webuse nlswork

. rwolf2 (xtreg wks_ue nev_mar i.year age, fe) (xtreg tenure nev_mar i.year age, fe) (reg hours nev_mar i.year), indepvars(nev_mar, nev_mar, nev_mar) seed(12011303) strata(occ_code)

Saved results

rwolf2 saves the following in e():

Scalars

e(rw_depvar1_indepvar1) The Romano Wolf p-value associated with indepdendent variable 1 in model 1, where
model 1 is indicated by the dependent variable name. Dependent variable--independent variable pair will be returned using both variables names.

...

e(rw_depvar2_indepvar1) The Romano Wolf p-value associated with indepdendent variable 1 in model 2, where
model 2 is indicated by the dependent variable name. Dependent variable--independent variable pair will be returned using both variables names.

...

Matrix

e(RW) The full set of Romano-Wolf corrected p-values, as well as the uncorrected p-values estimated by bootstrap and the baseline model, and Holm's p-value (if requested)

Acknowledgements

I am grateful to many users of rwolf for providing feedback and suggestions, and to David McKenzie for suggesting the alternative syntax implemented here.

References

Romano J.P. and Wolf M., 2005a.
Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing,
Journal of the American Statistical Association 100(469): 94-108.

Romano J.P. and Wolf M., 2005b.
Stepwise Multiple Testing as Formalized Data Snooping,
Econometrica 73(4): 1237-1282.

Romano J.P. and Wolf M., 2016.
Efficient computation of adjusted p-values for resampling-based stepdown multiple testing,
Statistics and Probability Letters 113: 38-40.

Clarke, D., Romano J.P. and Wolf M., 2020.
The Romano-Wolf Multiple Hypothesis Correction in Stata, Stata Journal 20(4): 812-843.

Clarke, D., 2021. rwolf2 Implementation and Flexible Syntax. Online.

Author

Damian Clarke, Department of Economics, University of Chile. dclarke@fen.uchile.cl