help wyoung

Title

wyoung -- Control the family-wise error rate when performing multiple hypothesis tests.

Syntax

Syntax 1: multiple hypothesis testing -- one model with multiple outcomes and subgroups

wyoung varlist, cmd(model) familyp(varlist) bootstraps(#)
[seed(#)
strata(varlist) cluster(varlist) force subgroup(varname)
controls("varlist1" ["varlist2" ...]) singlestep detail noresampling familypexp replace]

Syntax 2: multiple hypothesis testing -- different models with multiple outcomes and subgroups

wyoung, cmd("model1" ["model2" ...]) familyp("varname1" ["varname2" ...]) bootstraps(#)
[seed(#)
strata(varlist) cluster(varlist) force singlestep detail noresampling familypexp replace]

Options

cmd(), familyp()

Syntax 1: one model with multiple outcomes (see example 1 below)

cmd(model) specifies a single model with the multiple outcomes varlist. The outcome (dependent) variable is indicated in model by "OUTCOMEVAR" (upper case).
wyoung will estimate multiple outcome specifications by substituting each variable from varlist into "OUTCOMEVAR".

familyp(varlist) instructs wyoung to calculate adjusted p-values for the null hypotheses that the coefficients of varlist are equal to 0.

Syntax 2: different models with multiple outcomes (see example 2 below)

cmd("model1" ["model2" ...]) specifies a list of models.

familyp("varname1" ["varname2" ...]) instructs wyoung to calculate adjusted p-values for the null hypotheses that the coefficient of varname1 is equal to 0 in model1, the coefficient of varname2 is equal to 0 in model2, etc.
If only one varname is specified, wyoung applies it to all models.

bootstraps(#) performs # bootstrap replications for resampling. Westfall and Young (1993) recommend using at least 10,000 bootstraps.

seed(#) sets the random-number seed. Specifying this option is equivalent to typing the following command prior to calling wyoung:

. set seed #

strata(varlist) specifies variables identifying strata. If strata() is specified, bootstrap samples are selected within each stratum.

cluster(varlist) specifies variables identifying clusters.
If cluster() is specified, the sample drawn during each replication is a bootstrap sample of clusters.
This option is required if model includes clustered standard errors, unless force is specified.
See example 3 below.

force allows the user to include a model with clustered standard errors without also specifying the cluster() bootstrap option.

subgroup(varname) specifies an integer variable identifying subgroups.
If subgroup() is specified, wyoung will estimate models separately for each subgroup.
By default, specifying subgroup() will cause wyoung to select bootstrap samples within each subgroup, unless you specify otherwise in strata().
This option is only available when employing Syntax 1.
See example 4 below.

controls("varlist1" ["varlist2" ...]) lets you specify different controls for each outcome.
This option is only available when employing Syntax 1. The control variables are indicated in model by "CONTROLVARS" (upper case).
For the first outcome variable, wyoung will substitute varlist1 into "CONTROLVARS", for the second outcome it will
substitute varlist2, and so on. See example 7 below.

controlsinteract("varlist1" ["varlist2" ...]) is a variation on controls() that
will estimate the model separately for all pairwise combinations of outcome variables and specified controls.
Each set of controls will be substituted into "CONTROLVARS" as specified in model.
Specifying N different sets of controls (varlist1, varlist2, ..., varlistN) will multiply
the number of hypotheses being tested by N. See example 8 below.

singlestep computes the single-step adjusted p-value in addition to the step-down value. Resampling-based single-step methods often control type III (sign) error rates. Whether their
step-down counterparts also control the type III error rate is unknown (Westfall and Young 1993, p. 51).

detail produces sample size statistics for the bootstrap samples.

noresampling computes only the Bonferroni-Holm and Sidak-Holm adjusted p-values (very fast).

familypexp indicates that you are providing familyp(exp) instead of familyp(varlist) when employing Syntax 1, where exp specifies a coefficient or combination of coefficients.
exp follows the syntax of lincom and nlcom and must not contain an equal sign.
If employing Syntax 2, then familypexp indicates that
you are providing familyp("exp1" ["exp2" ...]) instead of familyp("varname1" ["varname2" ...]).
Specifying familypexp increases the set of possible hypothesis tests, but may cause wyoung to produce less helpful error messages when you make a syntax mistake.

replace replaces data in memory with wyoung results.

Description

wyoung calculates adjusted p-values using the free step-down resampling methodology of Westfall and Young (1993). It also computes the Bonferroni-Holm and Sidak-Holm adjusted p-values.

The family-wise error rate (FWER) is the probability of rejecting any true null hypothesis belonging to a "family" of hypotheses. In other words, the FWER is the probability of making at least one false discovery within the family.
When subset pivotality holds, the resampling-based method of Westfall and Young (1993) controls the FWER regardless of which hypotheses in the family happen to be true.

The sampling distribution of a pivotal statistic does not depend upon which distribution generated the data. The t-statistic is an example of a pivotal statistic.
Subset pivotality extends this concept to a multivariate setting. The subset pivotality condition requires that the multivariate distribution of any subvector of p-values is unaffected
by the truth or falsehood of hypotheses corresponding to p-values not included in the subvector. This condition is satisfied in many cases, including testing the significance of coefficients
in a general multivariate regression model with possibly non-normal distributions.

wyoung performs bootstrap resampling (i.e., sampling with replacement). It does not support permutation sampling (rerandomization).

Methods and formulas

The free step-down resampling method employed by wyoung is based on Algorithm 2.8 of Westfall and Young (1993).
The single-step resampling method (see option singlestep) is based on Algorithm 2.5 of Westfall and Young (1993). Detailed documentation
is available online at https://reifjulian.github.io/wyoung/documentation/wyoung.pdf.

The Bonferroni-Holm and Sidak-Holm step-down p-values are calculated as follows. Sort the J unadjusted p-values so that p(1)<p(2)<...<p(J).
The Bonferroni-Holm adjusted p-values are calculated as

.
The Sidak-Holm adjusted p-values are calculated as {1-(1-p(1))^J, max[p(1),1-(1-p(2))^(J-1)],..., max[p(J-1),p(J)]}.
If the calculation yields a value larger than 1, then the adjusted p-value is set equal to 1.

Following estimation of a model, wyoung obtains unadjusted p-values from r(table).
If r(table) is unavailable, then wyoung calculates a two-tailed p-value using the t-statistic and the residual degrees of freedom (as retrieved from e(df_r)).
If the residual degrees of freedom are unavailable, the unadjusted p-value is calculated by assuming normality.

Remarks

The cmd() option supports compound double quotes when employing Syntax 2. In this case,
the user should specify the list of models as
cmd(`" `"model1"' [`"model2"' ...] "').

The cmd() option allows the user to specify different estimation samples for each model when employing Syntax 2.
When specifying different samples, you may want to appropriately specify strata(varlist) to ensure balanced sample
sizes across bootstraps.

Citation

wyoung is not an official Stata command. It is a free contribution to the research community. You may cite it as:

Jones, D., D. Molitor, and J. Reif. 2019. "What Do Workplace Wellness Programs Do? Evidence from the Illinois Workplace Wellness Study." Quarterly Journal of Economics, November 2019, 134(4): 1747-1791.

Examples

1. Test whether the outcome variables mpg, headroom, or turn are significantly associated with displacement, conditional on length (3 hypotheses).

. sysuse auto.dta, clear

. wyoung mpg headroom turn, cmd(regress OUTCOMEVAR displacement length) familyp(displacement) bootstraps(100) seed(20)

2. Identical to example 1, but employs Syntax 2.

. wyoung, cmd("regress mpg displacement length" "regress headroom displacement length" "regress turn displacement length") familyp(displacement) bootstraps(100) seed(20)

3. Employ clustered standard errors.

. wyoung mpg headroom turn, cmd(regress OUTCOMEVAR displacement length, cluster(rep78)) cluster(rep78) familyp(displacement) bootstraps(100) seed(20)

4. Estimate models separately for the two subgroups defined by foreign (3 X 2 = 6 hypotheses).

. wyoung mpg headroom turn, cmd(regress OUTCOMEVAR displacement length) familyp(displacement) subgroup(foreign) bootstraps(100) seed(20)

5. Estimate models separately for the two subgroups defined by foreign, and also calculate adjusted p-values for length (3 X 2 X 2 = 12 hypotheses).

. wyoung mpg headroom turn, cmd(regress OUTCOMEVAR displacement length) familyp(displacement length) subgroup(foreign) bootstraps(100) seed(20)

6. Test the linear restriction _b[length] + 50*_b[displacement] = 0 (3 hypotheses).

. wyoung mpg headroom turn, cmd(regress OUTCOMEVAR displacement length) familyp(length+50*displacement) familypexp bootstraps(100) seed(20)

7. Estimate a model that uses different controls for two different outcomes (2 hypotheses).

Syntax 1:
. wyoung mpg rep78, cmd(regress OUTCOMEVAR displacement CONTROLVARS) familyp(displacement) controls("headroom" "turn") bootstraps(100) seed(20)

Syntax 2 (identical output):
. wyoung, cmd("regress mpg displacement headroom" "regress rep78 displacement turn") familyp(displacement) bootstraps(100) seed(20)

8. Estimate a model that interacts two different sets of controls with two different outcomes (4 hypotheses).

Syntax 1:
. wyoung mpg rep78, cmd(regress OUTCOMEVAR displacement CONTROLVARS) familyp(displacement) controlsinteract("headroom" "turn") bootstraps(100) seed(20)

Syntax 2 (identical output):
. wyoung, cmd("regress mpg displacement headroom" "regress rep78 displacement headroom" "regress mpg displacement turn" "regress rep78 displacement turn") familyp(displacement) bootstraps(100) seed(20)

Stored results

Specifying the replace option will replace data in memory with wyoung results.
In addition, wyoung stores the following in r():

Macros

r(cmd) wyoung

r(cmdline) command as typed

Matrices

r(table) matrix of results

References

Jones, D., D. Molitor, and J. Reif. "What Do Workplace Wellness Programs Do? Evidence from the Illinois Workplace Wellness Study." Quarterly Journal of Economics, November 2019, 134(4): 1747-1791.
Available from: www.nber.org/workplacewellness/s/IL_Wellness_Study_1.pdf.

Westfall, P., and S. Young. 1993. Resampling-based Multiple Testing: Examples and Methods for p-value Adjustment. John Wiley & Sons, Inc.

Author

Julian Reif, University of Illinois

jreif@illinois.edu

Also see

test,
anova