Heterogeneity tests and inverse-variance weights in parmest resultssets
parmhet est_varname se_varname [ dof_varname ] [if] [in] [ , basic_opts resultsset_opts hettest_opts ]
parmiv est_varname se_varname [ dof_varname ] [if] [in] [ , basic_opts hettest_opts ]
Description
parmhet and parmiv are designed for use with parmest resultssets, which have one observation per estimated parameter and data on parameter estimates. parmhet inputs variables containing parameter estimates, standard errors and (optionally) degrees of freedom, and outputs an output dataset (or resultsset) with one observation, or one observation per by-group, and data on heterogeneity test statistics on the input parameters in the dataset or by-group. The output dataset can be listed to the Stata log, saved to a disk file, or written to memory, overwriting the existing dataset. Optionally, parmhet may also add variables to the existing dataset, containing inverse-variance weights and/or semi-weights and/or semi-weight-based standard errors. These additional variables may then be input to the metaparm module of the parmest package, to produce summary meta-analysed parameter estimates, using the fixed-effect method or the DerSimonian-Laird randomly-variable-effect method. parmiv is a routine intended for use by programmers, and inputs variables containing estimates, standard errors and (optionally) degrees of freedom, and outputs additional variables in the existing dataset, containing inverse-variance weights and/or semi-weights and/or semi-weight-based standard errors and/or heterogeneity test statistics for the dataset or by-group.
Options
parmhet and parmiv have a large number of options, which fall into the following 3 groups:
Option group Description -------------------------------------------------------------------------- basic_opts Basic options for parmhet and parmiv resultsset_opts Resultsset options for parmhet hettest_opts Heterogeneity-test options for parmhet and parmiv --------------------------------------------------------------------------
Basic options for parmhet and parmiv
options Description ------------------------------------------------------------------------- by(varlist) Variables specifying by-groups eform Estimates and confidence limits exponentiated float Numeric output variables of type float or less dfcombine(combination_rule) Rule for combining degrees of freedom ivweight(newvarname) Name of generated inverse-variance weight variable sweight(newvarname) Name of generated semi-weight variable sstderr(newvarname) Name of generated semi-weight-based standard error variable -------------------------------------------------------------------------
where combination_rule is
welch | constant
See parmhet_basic_opts for details of these options.
Resultsset options for parmhet
options Description ------------------------------------------------------------------------- list(list_spec) List output dataset to Stata log and/or Results window saving(filename[,replace]) Save output dataset to a disk file norestore Write output dataset to memory fast Write output dataset to memory without precautions flist(global_macro_name) Append output filename to a global macro idnum(#) Value of numeric dataset ID variable nidnum(newvarname) Name of numeric dataset ID variable idstr(string) Value of string dataset ID variable nidstr(newvarname) Name of string dataset ID variable sumvar(varlist) Variables to be summed in the output dataset format(formatting_list) Display formats for variables in the output dataset keep(varlist) Variables to be kept in the output dataset -------------------------------------------------------------------------
where formatting_list is a list of form
varlist_1 format_1 ... varlist_n format_n
and list_spec is a specification of the form
[varlist] [if] [in] [ , [list_options ] ]
and list_options is a list of options used by the list command.
See parmhet_resultsset_opts for details of these options.
Heterogeneity-test options for parmhet and parmiv
options Description ------------------------------------------------------------------------- chi2het(newvarname) Heterogeneity chi-squared statistic variable dfhet(newvarname) Heterogeneity degrees of freedom variable i2het(newvarname) Heterogeneity I-squared statistic variable tau2het(newvarname) Heterogeneity tau-squared statistic variable fhet(newvarname) Heterogeneity F-statistic variable resdfhet(newvarname) Heterogeneity residual degrees of freedom variable phet(newvarname) Heterogeneity P-value variable -------------------------------------------------------------------------
See parmhet_hettest_opts for details of these options.
Output dataset created by parmhet
This output dataset (or resultsset) is described in detail in parmhet_resultsset.
Methods and formulas
parmhet and parmiv use formulas introduced by Cochran (1954) and Welch (1951), and later extended by DerSimonian and Laird (1986) and by Higgins and Thompson (2002). In general, it is assumed that, in the input dataset (or in a by-group of the input dataset), there are N observations, corresponding to parameters theta_1, ..., theta_N, with corresponding standard errors se_1, ..., se_N, and corresponding degrees of freedon df_1, ..., df_N if a degrees of freedom variable is specified. The parameters theta_j and the standard errors se_j are given by the input variables specified by est_varname and se_varname, unless the eform option is specified, in which case the theta_j are the logs of the values of est_varname, and the se_j are the standard errors of these logs. We assume that the theta_j were sampled independently, or at least in an uncorrelated way, from N respective sub-populations.
The inverse-variance weights, output in the ivweight() variable, are defined by the formula
ivweight_j = (se_j)^(-2)
for each j from 1 to N. The inverse-variance-weighted mean parameter value is
Theta = Sum ( iwweight_j * theta_j ) / Sum (ivweight_j)
where Sum() is the sum for j from 1 to N. Theta can be used as an estimate for a common sub-population mean parameter value, assuming that there is no heterogeneity between sub-populations. This estimate, with a standard error, confidence limits and a P-value, can be calculated by inputting the estimates, standard errors and (optionally) degrees of freedom to the metaparm option of the parmest package, with aweights equal to the ivweight() variable.
The heterogeneity chi-squared statistic is defined by
Q = Sum ( ivweight_j * (theta_j - Theta)^2 )
Under the null hypothesis of no heterogeneity, Q is sampled from a distribution approximating to a chi-squared distribution with N-1 degrees of freedom. The I-squared statistic of Higgins and Thompson (2002) is equal to
I^2 = 100*max( 0 , (Q-N+1)/(N-1) )
and is equal to zero if the heterogeneity chi-squared statistic is no more than its mean of N-1 under the null hypothesis of no heterogeneity, and otherwise measures the percent relative excess of heterogeneity, compared to this null mean. And the tau-squared statistic of Higgins and Thompson (2002) is given by
tau^2 = max( 0 , (Q-N+1)/( Sum(ivweight_j) - (Sum((ivweight_j)^2))/(Sum(ivweight_j)) ) )
and estimates the sampling variance of the true sub-population values of the theta_j in the meta-population from which their respective sub-populations were sampled. It is expressed in squared theta_j units.
The sampling variance of each theta_j, in the two-stage sampling process where the jth sub-population is sampled from the meta-population and the theta_j is sampled from the jth subpopulation, can be estimated by (se_j)^2 + tau^2. It follows that the two-stage sampling standard error of theta_j, output in the sstderr() variable, is equal to
sstderr_j = sqrt((se_j)^2 + tau^2)
and the two-stage inverse-variance weights (or semi-weights), output in the sweight() variable, is equal to
sweight_j = ((se_j)^2 + tau^2)^{-1}
These two-stage sampling standard errors and two-stage inverse-variance weights can then be input to the metaparm module of the parmest package, to produce an estimate, a standard error, confidence limits, and a P-value for the common meta-population mean of the theta_j. The sstderr() variable generated by parmhet or parmiv is input to metaparm as the stderr() option, and the sweight() variable generated by parmhet or parmiv is input to metaparm as the aweights. This computation implements the randomly-variable-effect method of DerSimonian and Laird (1986).
An alternative to fixed-effect meta-analysis and randomly-variable-effect meta-analysis is nonrandomly-variable-effect meta-analysis. This can be implemented using metaparm by setting the aweights equal to the respective sample numbers used in estimating the theta_j. The nonrandomly-variable-effect method does not assume that the sub-population means of the theta_j are equal (as the fixed-effect method does), and does not assume that we can estimate the meta-population variance (as the randomly-variable-effect method does). However, it does assume that the sample-size-weighted average of the sub-population effects is a useful parameter to know, even thought the sub-population effects may vary between sub-populations. Whichever of the three methods we use, our colleagues will usually expect a heterogeneity test to be done.
The F-test statistic, computed if a degrees of freedom variable is supplied to parmhet or parmiv, is defined by a method specified by the dfcombine() option. If the user specifies dfcombine(welch) (the default), then the F-test statistic, and its denominator degrees of freedom, are defined using the formulas of Welch (1951), popularized by Cochran (1954). If the user specifies dfcombine(constant), then parmhet and parmiv check that the input degrees of freedom variable is constant, or constant within by-groups if a by() option is specified, and then defines the F-statistic as Q/(N-1), with N-1 numerator degrees of freedom, and the denominator degrees of freedom given by the constant value of the input degrees of freedom variable. The option dfcombine(constant) is useful if the input parameters are uncorrelated parameters of a single regression model, and their standard errors are computed using an equal-variance formula, and the degrees of freedom is pooled for all parameters in the model.
Remarks
parmhet and parmiv are designed for use with the parmest package, and particularly with the metaparm module of that package. More information about the production and use of parmest resultssets can be found in Newson (2010), Newson (2008), Newson (2006), Newson (2004), Newson (2003) and Newson (2002). parmhet can be used with parmest, metaparm, eclplot and listtab to form a comprehensive system for meta-analyses and/or interaction analyses, enabling the user to produce plots and/or tables of results.
Other programs are available in the Stata community for carrying out meta-analyses, notably metan, written by Michael J. Bradburn, Jonathan J. Deeks, Douglas G. Altman, Ross J. Harris, Roger M. Harbord, and Jonathan A. C. Sterne. (See meta and [R] meta for details.) The combination of parmhet, parmest, metaparm, eclplot and listtab is arguably even more flexible than metan, at the price of having to know some Stata programming.
Other programs for heterogeneity tests on parmest resultssets are the estparm and estparmtest modules of the estparm package. These allow the user to carry out Wald heterogeneity tests, comparing each other parameter estimate with the first parameter estimate. The Cochran and Welch heterogeneity tests, by contrast, weight the parameters inversely by variance, and do not otherwise treat the first parameter differently from the others. It is therefore expected that parmhet will supersede estparm for most purposes.
The parmest, eclplot, listtab, estparm and metan packages can be downloaded from SSC.
Examples: Sequence 1
The first example sequence uses the dataset parmhet_example1, which is distributed with the parmhet package as an ancillary file. This dataset contains data from a meta-analysis of Glasziou et al. (1993), distributed in Bland (2000), and involving 5 clinical trials, comparing death rates in subjects treated with vitamin A and untreated control subjects. The dataset contains 1 observation per treatment group per trial, and data on total numbers of subjects, and numbers of deaths, in that treatment group in that trial, and also data on the quantity (in international units or IU) and frequency (per unit time) of the vitamin A dosage applied to the treated group in that trial. The set-up uses the parmby module of the parmest package, downloadable from SSC, to replace this dataset with a new dataset (or resultsset) in memory, with 1 observation per trial, and data on odds ratios of death with respect to vitamin A treatment, with their confidence limits and P-values, and also the total number of subjects in the trial (in variable N).
Set-up
. use "http://www.imperial.ac.uk/nhli/r.newson/stata10/parmhet_example1 > .dta", clear . describe . parmby "blogit deaths number vitamina, or", by(studyseq doseiu dosefreq) eform norestore escal(N) rename(es_1 N) . keep if parm=="vitamina" . describe . list studyseq doseiu dosefreq parm N estimate stderr min* max* p
Simple examples
. parmhet estimate stderr, eform list(,)
. parmhet estimate stderr, eform by(doseiu) list(, subvarname abbr(32))
. parmhet estimate stderr, eform by(doseiu) saving(myhet1.dta, replace)
Note that, if we do separate heterogeneity tests for different vitamin A doses using the option by(doseiu), then dosage groups with only 1 study have zero heterogeneity chi-squared statistics, zero heterogeneity degrees of freedom, zero I-squared and tau-squared statistics, and missing heterogeneity P-values. Note, also, that the suboptions subvarname abbr(32) of the list() option cause the heterogeneity test statistics to be listed with informative headings. These are set by the varname characteristic, set by parmhet and used by the list command.
The following example uses the norestore option. This replaces the resultssetset in memory, containing 1 observation per study, with a new resultsset, containing 1 observation per vitamin A dosage group, and data on heterogeneity test results:
. preserve . parmhet estimate stderr, eform by(doseiu) norestore . describe . list . restore
The following example uses parmhet together with the metaparm module of the parmest package. parmhet tests for heterogeneity, and also adds to the existing dataset a new variable ivwt, containing inverse-variance weights for each study. These are then used as aweights by metaparm to perform a fixed-effect meta-analysis, estimating a common odds ratio for all studies, and assuming that such a common odds ratio exists.
. parmhet estimate stderr, eform list(,) ivweight(ivwt) . metaparm [aweight=ivwt], eform sumvar(N) list(,)
The following example uses parmhet with metaparm to perform a DerSimonian-Laird randomly-variable-effects meta-analysis (DerSimonian and Laird, 1986). This time, parmhet adds variables semiwt and swse, containing semi-weights and semi-weight-based standard errors, respectively. These are passed to metaparm as aweights and standard errors, respectively.
. parmhet estimate stderr, eform list(,) sweight(semiwt) sstderr(swse) . metaparm [aweight=semiwt], eform sumvar(N) stderr(swse) list(,)
The following example performs a nonrandomly-variable-effects analysis, weighted by the study size variable N.
. parmhet estimate stderr, eform sumvar(N) list(,) . metaparm [aweight=N], eform sumvar(N) list(,)
The following example carries out a nonrandomly-variable-effects meta-analysis, saving the resultssets from parmhet and metaparm in disk-file datasets myhet and mymeta, respectively. These resultssets are appended to the existing resultsset to produce an extended resultsset, which is listed to show the study odds ratios, the weighted geometric mean (GM) odds ratio, and P-values for the odds ratios and for the heterogeneity test.
. preserve . gene idstr="Single study" . order idstr . parmhet estimate stderr, eform sumvar(N) idstr(Heterogeneity) phet(p) keep(idstr N p) saving(myhet.dta, replace) . metaparm [aweight=N], eform sumvar(N) idstr("Weighted GM") saving(mymeta.dta, replace) . append using mymeta . append using myhet . describe . list idstr studyseq doseiu dosefreq N estimate min* max* p, sepby(idstr) . restore
Examples: Sequence 2
The following examples also use the dataset parmhet_example1. However, this time, the parmby module of parmest is used to produce a dataset of log odds ratios. These are then tested for heterogeneity, using parmhet without the eform option.
Set-up
. use "http://www.imperial.ac.uk/nhli/r.newson/stata10/parmhet_example1 > .dta", clear . describe . parmby "blogit deaths number vitamina", by(studyseq doseiu dosefreq) norestore escal(N) rename(es_1 N) . keep if parm=="vitamina" . describe . list studyseq doseiu dosefreq parm N estimate stderr min* max* p
Simple examples
. parmhet estimate stderr, list(,)
. parmhet estimate stderr, by(doseiu) list(,)
Examples: Sequence 3
The following example uses the auto data, distributed with official Stata, and adds a new variable continent, containing the continent of origin of a car model. The parmby module of parmest is then used to create a resultsset with 1 parameter per continent and data on mean weights of cars (in US pounds), with numbers of car models, estimates, standard errors, degrees of freedom, confidence limits, and P-values. This resultsset is then input to parmhet to perform a Welch unequal-variance heterogeneity test (Welch, 1951), which shows that mean car weights vary between continents.
Set-up
. sysuse auto, clear . gene firm=word(make,1) . lab var firm "Firm" . lab def continent 1 "America" 2 "Asia" 3 "Europe" . gene continent=1 if !foreign . replace continent=2 if inlist(firm,"Datsun","Honda","Mazda","Subaru","Toyota") . replace continent=3 if inlist(firm,"Audi","BMW","VW","Fiat","Peugeot","Renault","Volvo") . lab val continent continent . lab var continent "Continent of origin" . describe . tab firm continent, missing . parmby "mean weight", by(continent) norestore . describe . list, abbr(32)
Examples
. parmhet estimate stderr dof, list(,)
. parmhet estimate stderr dof, list(, subvarname abbr(32))
Author
Roger Newson, National Heart and Lung Institute, Imperial College London, UK. Email: r.newson@imperial.ac.uk
References
Bland, M. 2000. An introduction to medical statistics. 3rd ed. Oxford: Oxford University Press.
Cochran, W. G. 1954. The combination of estimates from different experiments. Biometrics 10(1): 101-129.
DerSimonian, R. and Laird, N. 1986. Meta-analysis in clinical trials. Controlled Clinical Trials 7(3): 177-188.
Glasziou, P.P. and Mackerras, D. E. M. 1993. Vitamin A supplementation in infectious disease: a meta–analysis. British Medical Journal 306(6874): 366-370.
Higgins, J. P. T. and Thompson, S. G. 2002. Quantifying heterogeneity in a meta-analysis. Statistics in Medicine 21(11): 1539-1558.
Newson, R. B. 2010. Post-parmest peripherals: fvregen, invcise, and qqvalue. Presented at the 16th United Kingdom Stata Users' Group Meeting, London, 9-10 September, 2010. Also downloadable from Roger Newson's website at http://www.imperial.ac.uk/nhli/r.newson.
Newson, R. B. 2008. parmest and extensions. Presented at the 14th United Kingdom Stata Users' Group Meeting, London, 8-9 September, 2008. Also downloadable from Roger Newson's website at http://www.imperial.ac.uk/nhli/r.newson.
Newson, R. 2006. Resultssets, resultsspreadsheets, and resultsplots in Stata. Presented at the 4th German Stata Users' Group Meeting, Mannheim, 31 March, 2006. Also downloadable from Roger Newson's website at http://www.imperial.ac.uk/nhli/r.newson.
Newson, R. 2004. From datasets to resultssets in Stata. Presented at the 10th United Kingdom Stata Users' Group Meeting, London, 29-30 June, 2004. Also downloadable from Roger Newson's website at http://www.imperial.ac.uk/nhli/r.newson.
Newson, R. 2003. Confidence intervals and p-values for delivery to the end user. The Stata Journal 3(3): 245-269. Pre-publication draft downloadable from Roger Newson's website at http://www.imperial.ac.uk/nhli/r.newson.
Newson, R. 2002. Creating plots and tables of estimation results using parmest and friends. Presented at the 8th United Kingdom Stata Users' Group Meeting, 20-21 May, 2002. Also downloadable from Roger Newson's website at http://www.imperial.ac.uk/nhli/r.newson.
Welch, B. L. 1951. On the comparison of several mean values: an alternative approach. Biometrika 36(3/4): 330-336.
Also see
Manual: [R] meta, [R] test On-line: help for parmhet_basic_opts, parmhet_resultsset_opts, parmhet_hettest_opts, parmhet_resultsset help for test, list help for parmest, parmby, parmcip, metaparm, eclplot, listtab, estparm, metan if installed