*! version 1.0.0 07Sep2020

Title

metastrong -- Estimate the proportion of true effect sizes above or below a threshold in random-effects meta-analysis

Syntax

metastrong
#q
[,
exp
above
parametric
bootstrap[(bootstrap_options)] ]

q represents the true effect size that is the threshold for "scientific importance"

Before using metastrong, the data must be declared with meta set or meta esize

option

exp specify that q is exponentiated

above estimate the proportion of effects above q; default is the proportion of effects below q

parametric compute the proportion of true effects above (or below) q using the parametric method; default is to
calibrate estimates and then compute proportions non-parametrically

bootstrap[(bootstrap_options)] specify bootstrap options for estimating the
proportion (and CIs) of true effects above (or below) . All bootstrap options are available

[if][in] qualifiers can be specified when meta setting the data using meta set or meta esize.

Description

metastrong estimates evidence strength for scientifically meaningful effects in meta-analyses under effect heterogeneity
(ie, a nonzero estimated variance of the true effect distribution) as proposed by Mathur and VanderWeele (2019; 2020). metastrong
reports the estimated proportion of true effect sizes above or below a chosen threshold of a meaningful effect size q, together with confidence
intervals derived via the bootstrap.

These metrics could help identify if (1) there are few effects of scientifically meaningful size despite a “statistically
significant” pooled point estimate, (2) there are some large effects despite an apparently null point estimate, or (3)
strong effects in the direction opposite the pooled estimate also regularly occur (and thus, potential effect modifiers
should be examined) (Mathur and VanderWeele 2019).

By default, metastrong performs estimation using the "calibrated" method (Mathur and VanderWeele 2020) that extends work by Wang and Lee (2019).
metastrong first calibrates the effect estimates and then computes the proportion of studies above (or below) q. This method makes no assumptions
about the distribution of true effects and performs well in meta-analyses with as few as 10 studies.

When the parametric option is specified, metastrong estimates the proportion of studies above (or below) q using the formulae devised
by Mathur and VanderWeele (2019). This estimate is then bootstrapped to derive confidence intervals. As with the calibrated method, at least 10 individual
studies should be available in the meta-analysis for these estimates to provide valid results.

Point estimates produced by metastrong correspond to those produced by the function prop_stronger in the R package MetaUtility as of version 2.0.0.
Confidence intervals will likely differ between packages given that Stata and R use different random number seeds for determining which studies to include
in each bootstrap repetition.

Options

Main

exp specifies that q is exponentiated. For example, if q is 0.70 and represents a risk-ratio, then the user must specify exp. Conversely,
if the user specifies that q is -.35667494 (the log of 0.70), then exp should not be specified.

above specifies that the proportion of effects above q should be estimated. The default is to estimate the proportion of effects below q.

parametric specifies that the proportion of true effects above (or below) q should be estimated using the parametric method. The default is to
calibrate all effect estimates and then compute proportions non-parametrically.

bootstrap[(bootstrap_options)] specifies bootstrap options for estimating confidence intervals.
All bootstrap options are available.

Examples

1) metastrong with a binary outcome:

Load example data

. webuse bcgset, clear

Use meta esize to compute effect sizes for the log risk-ratio using a random effects(REML) model

. meta esize npost nnegt nposc nnegc, esize(lnrratio) studylabel(studylbl)

Generate a forest plot to review the individual studies and pooled estimates

. meta forestplot, eform nullrefline

We now use metastrong to estimate the proportion of true effect sizes above a threshold of 0.70 (risk-ratio) using the parametric method.
Additionally, we specify that that bootstrap should execute 1000 repetitions and employ the BCA method.

. metastrong 0.70, exp above parametric boot(reps(1000) bca)

Same as before, but we now estimate the proportion of true effect sizes below the same threshold of 0.70 (risk-ratio).

. metastrong 0.70, exp below parametric boot(reps(1000) bca)

We review all bootstrap estimates.

. estat boot, all

We now estimate the proportion of true effect sizes above a threshold of 0.70 (risk-ratio) using the "calibrated" method.
We do not specify that that bootstrap should employ the BCA method because it will produce an error.

. metastrong 0.70, exp above boot(reps(1000))

We review all bootstrap estimates.

. estat boot, all

2) metastrong with a continuous outcome:

Load example data

. use https://www.stata-press.com/data/r16/pupiliq, clear

meta set the data

. meta set stdmdiff se, studylabel(studylbl) eslabel(Std. Mean Diff.)

meta summarize the data to review the individual studies and pooled estimates

. meta summarize

We use metastrong to estimate the proportion of true effect sizes above a threshold of 0.20 (considered a small effect on the standardized mean difference scale)
using the parametric method. We also specify that that bootstrap should execute 1000 repetitions and employ the BCA method.

. metastrong 0.20, above parametric boot(reps(1000) bca)

We now use the "calibrated" method. We do not specify that that bootstrap should employ the BCA method because it will produce an error.

. metastrong 0.20, above boot(reps(1000))

We review all bootstrap estimates.

. estat boot, all

Acknowledgments

I thank John Moran for advocating that I write this package.

Stored results

metastrong stores the following in r():

Scalars

r(phat) estimated proportion of studies above (or below) threshold q

References

Mathur, M. B. and T. J. VanderWeele. 2019. New metrics for meta-analyses of heterogeneous effects. Statistics in Medicine 38: 1336–1342.

Mathur, M. B. and T. J. VanderWeele. 2020. Robust metrics and sensitivity analyses for meta-analyses of heterogeneous effects. Epidemiology 31: 356-358.

Wang, C. C. and W. C. Lee. 2019. A simple method to estimate prediction intervals and predictive distributions: Summarizing meta‐analyses beyond means and confidence intervals.
Research Synthesis Methods 10: 255-266.

Citation of metastrong

metastrong is not an official Stata command. It is a free contribution
to the research community, like a paper. Please cite it as such:

Linden A. (2020). METASTRONG: Stata module for estimating the proportion of true effect sizes above or below a threshold in random-effects meta-analysis.

Authors

Ariel Linden

President, Linden Consulting Group, LLC

alinden@lindenconsulting.org

Also see

Online: meta, meta set, meta regress, metafrag (if installed)