{…}

help for caterpillar

Title

caterpillar - Given a set of estimates and standard errors, generate confidence intervals, Bonferroni-corrected confidence intervals, and null distribution.

Syntax

caterpillar est se id [if] [in], [saving(dataset) by(id) options]

options

center If specified, centers estimates around their precision-weighted mean and creates new variable, contrast.

graph If specified, creates graph of estimates (centered if center is also used), 95% pointwise confidence intervals, Bonferroni-corrected confidence intervals, and null distribution. Cannot be used with -by-.

Description

caterpillar takes a set of estimates est and standard errors se, along with a unique identifier id for each estimate.
The estimates may represent the effects of different programs (as in von Hippel & Bellows, 2017) or the results of different studies (as in a meta-analysis).

caterpillar outputs a "caterpillar plot" containing point estimates (sorted in ascending order), along with 95% pointwise confidence intervals, Bonferroni-corrected 95% confidence intervals, and an estimate of the null distribution.
The null distribution represents what the distribution of estimates would look like under homogeneity -- i.e., if there were no differences between the effects, and the estimates differed only because of random estimation error.

caterpillar prints summary statistics on the plot, including Cochran's Q test (with degrees of freedom and p value), a method-of-moments estimate of the heterogeneity standard deviation (tau), and the Higgins-Thompson estimate of the reliability (rho). All calculations described in von Hippel and Bellows (2018).

The program generates five variables, CI_lo, CI_hi, CI_lo_bon, CI_hi_bon, and null_quantile, that contain the confidence intervals and quantiles of the null distribution.

caterpillar optionally saves Cochran's Q test, degrees of freedom for Cochran's Q, p value for Cochran's Q, the heterogeneity standard deviation, and the reliability into a separate dataset.

Remarks

caterpillar requires that _gwtmean be installed from SSC.

caterpillar requires a unique identifier for estimates and will not complete if a unique identifier is not given. If the -by- option is used, caterpillar requires that the identifier is unique within each group.

Examples

This example uses the dataset caterpillar_replication.dta to replicate some results from von Hippel & Bellows (2018).
To replicate von Hippel & Bellows more completely, run the code in vonHippelBellows2018.do.

Use install caterpillar_replication.dta and vonHippelBellows2018.do in the working directory, run {}cmd:ssc install caterpillar, all}.

. // Graph CIs, Bonferroni-corrected CIs, and null distribution
. // for all New York City teacher preparation programs (TPPs) in math
. use caterpillar_replication, clear
. caterpillar est se tpp if state=="NYC" & subject=="Math" ///
. & size=="All", graph center

. // Re-graph without the outlier.
. // The Q test becomes non-significant, and
. // the heterogeneity and reliability estimates go to 0.
. use caterpillar_replication, clear
. caterpillar est se tpp if state=="NYC" & subject=="Math" ///
. & size=="All" & est<.4, graph center

. // Generate the same statistics (except for graph) separately for different states and test subjects (math, reading, etc.),
. // as well as different data subsets and modeling decisions described in von Hippel & Bellows (2018).
. use caterpillar_replication, clear
. caterpillar est se tpp, by(state subject size schlfe experienced se_inflation) ///
. center saving(tpp_estimates, replace)

Saved Results

Scalars:

r(Q) Cochran's Q statistic

r(df) Degrees of freedom for Cochran's Q statistic

r(p) P-value for Cochran's Q statistic

r(tau) Heterogeneity standard deviation

r(rho) Reliability

Matrices:

If -by- is used, instead of scalars, statistics are saved into vectors. Also, a local macro r(levels) holds the -by- levels for reference.

r(Q) Vector of Cochran's Q statistics, by group

r(df) Vector of degrees of freedom for Cochran's Q statistic, by group

r(p) Vector of p-values for Cochran's Q statistic, by group

r(tau) Vector of heterogeneity standard deviation, by group

r(rho) Vector of reliabilities, by group

Authors

Laura Bellows, Duke University, USA

laura.bellows@duke.edu

Paul von Hippel, University of Texas at Austin, USA

paulvonhippel.utaustin@gmail.com

References

von Hippel, Paul T., and Laura Bellows (2018). "How Much Does Teacher Quality Vary Across Teacher Preparation Programs? Reanalyses from 6 States." Economics of Education Review, in press.
Also available as SSRN Working Paper, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2990498.

Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558. https://doi.org/10.1002/sim.1186

Also see

von Hippel, Paul T., Laura Bellows, Cynthia Osborne, Jane Lincove, and Nicholas Mills (2016). "Teacher Quality Differences Between Teacher Preparation Programs: How Big? How Reliable? Which Programs Are Different?" Economics of Education Review 53, 31-45. doi:10.1016/j.econedurev.2016.05.002