*! ml_fragility.sthlp metaLong for Stata 14.1

ml_fragility — Leave-k-out Fragility Analysis

Syntax

ml_fragility yi vi [if] [in] ,
study(varname)
time(varname)
metafile(filename)
[alpha()
maxk()
nosmallsample
saving()
replace]

Description

ml_fragility computes the fragility index at each follow-up time point.
Studies are removed one at a time (leave-one-out) or in random combinations of
size 2 to maxk() (leave-k-out). The fragility index is the minimum
number of removals required to flip the significance conclusion (significant →
non-significant, or vice versa).

A fragility index of 1 means a single study's removal changes the conclusion.
The fragility quotient (index / k) normalises for sample size.

Required options

study(varname) specifies the study (cluster) identifier variable.

time(varname) specifies the numeric follow-up time variable.

metafile(filename) specifies the path to the results dataset
saved by ml_meta.

Main options

alpha() sets the significance level. Default is 0.05.

maxk() sets the maximum number of studies to remove.
Default is 5. Larger values are more exhaustive but considerably slower.

nosmallsample uses z-based (large-sample) inference in the re-fit
models instead of t(k−1).

saving() saves the fragility results to filename.dta.

replace allows overwriting an existing saving() file.

Saved dataset columns

time Follow-up time

k_studies Number of unique studies at this time point

p_original Original p-value from ml_meta

sig_original 1 if originally significant at alpha

fragility_index Min removals to flip significance;
missing if FI > maxk

frag_quotient fragility_index / k_studies

study_removed Study ID whose removal flipped (leave-one-out only)

Returned r() values

r(fragility) Matrix of results

r(alpha) Significance level used

r(maxk) Maximum k tested

Algorithm

At each time point t with k studies:

Step 1 — Leave-one-out:
For each study s, remove it, re-estimate the pooled effect with DL tau2 and
cluster-robust SE, and check whether significance flips. If any single removal
flips significance, fragility_index = 1.

Step 2 — Leave-k-out (k = 2 … maxk):
If the LOO pass found no flip, up to 500 random combinations of k studies are
tested for each k. The smallest k producing a flip defines the fragility index.

Note on computation time

For large datasets or many time points with many studies, computation can be
slow. Reducing maxk() or restricting to specific time points with if
will accelerate the analysis.

Example

. sim_longmeta, k(12) times(0 6 12) seed(5) clear

. ml_meta yi vi, study(study) time(time) saving(meta_res) replace

. ml_fragility yi vi, study(study) time(time) ///

metafile(meta_res) maxk(3) alpha(0.05) saving(frag_res) replace

. use frag_res, clear

. list time k_studies fragility_index frag_quotient study_removed

References

Walsh, M., et al. (2014). The statistical significance of randomized controlled
trial results is frequently fragile: a case for a Fragility Index.
Journal of Clinical Epidemiology, 67(6), 622–628.

Hedges, L.V., Tipton, E., & Johnson, M.C. (2010).
Robust variance estimation in meta-regression with dependent effect sizes.
Research Synthesis Methods, 1(1), 39-65.

See also

ml_meta, ml_sens, metalong_plot, metalong