{smcl} {* 4 January 2018}{…} {hline} help for {hi:caterpillar} {hline} {title:Title} {pstd}{cmd:caterpillar} - Given a set of estimates and standard errors, generate confidence intervals, Bonferroni-corrected confidence intervals, and null distribution. {title:Syntax} {pstd}{cmd:caterpillar} {it:est} {it:se} {it:id} {ifin}, [saving(dataset) by(id) {it: options}] {synoptset 25 tabbed}{...} {marker opt}{synopthdr:options} {synoptline} {synopt :{opt center}} If specified, centers estimates around their precision-weighted mean and creates new variable, {it:contrast}. {p_end} {synopt :{opt graph}} If specified, creates graph of estimates (centered if {it:center} is also used), 95% pointwise confidence intervals, Bonferroni-corrected confidence intervals, and null distribution. Cannot be used with -by-. {p_end} {title:Description} {pstd}{cmd:caterpillar} takes a set of estimates {it:est} and standard errors {it:se}, along with a unique identifier {it:id} for each estimate. The estimates may represent the effects of different programs (as in von Hippel & Bellows, 2017) or the results of different studies (as in a meta-analysis). {pstd}{cmd:caterpillar} outputs a "caterpillar plot" containing point estimates (sorted in ascending order), along with 95% pointwise confidence intervals, Bonferroni-corrected 95% confidence intervals, and an estimate of the null distribution. The null distribution represents what the distribution of estimates would look like under homogeneity -- i.e., if there were no differences between the effects, and the estimates differed only because of random estimation error. {pstd}{cmd:caterpillar} prints summary statistics on the plot, including Cochran's Q test (with degrees of freedom and p value), a method-of-moments estimate of the heterogeneity standard deviation (tau), and the Higgins-Thompson estimate of the reliability (rho). All calculations described in von Hippel and Bellows (2018). {pstd}The program generates five variables, {it:CI_lo}, {it:CI_hi}, {it:CI_lo_bon}, {it:CI_hi_bon}, and {it:null_quantile}, that contain the confidence intervals and quantiles of the null distribution. {pstd}{cmd:caterpillar} optionally saves Cochran's Q test, degrees of freedom for Cochran's Q, p value for Cochran's Q, the heterogeneity standard deviation, and the reliability into a separate dataset. {title:Remarks} {pstd}{cmd:caterpillar} requires that {cmd:_gwtmean} be installed from SSC. {pstd}{cmd:caterpillar} requires a unique identifier for estimates and will not complete if a unique identifier is not given. If the -by- option is used, {cmd:caterpillar} requires that the identifier is unique within each group. {title:Examples} {pstd}This example uses the dataset {it:caterpillar_replication.dta} to replicate some results from von Hippel & Bellows (2018). To replicate von Hippel & Bellows more completely, run the code in {it:vonHippelBellows2018.do}. Use install {it:caterpillar_replication.dta} and {it:vonHippelBellows2018.do} in the working directory, run {}cmd:ssc install caterpillar, all}. . // Graph CIs, Bonferroni-corrected CIs, and null distribution . // for all New York City teacher preparation programs (TPPs) in math . use caterpillar_replication, clear . caterpillar est se tpp if state=="NYC" & subject=="Math" /// . & size=="All", graph center . // Re-graph without the outlier. . // The Q test becomes non-significant, and . // the heterogeneity and reliability estimates go to 0. . use caterpillar_replication, clear . caterpillar est se tpp if state=="NYC" & subject=="Math" /// . & size=="All" & est<.4, graph center . // Generate the same statistics (except for graph) separately for different states and test subjects (math, reading, etc.), . // as well as different data subsets and modeling decisions described in von Hippel & Bellows (2018). . use caterpillar_replication, clear . caterpillar est se tpp, by(state subject size schlfe experienced se_inflation) /// . center saving(tpp_estimates, replace) {title:Saved Results} {pstd}Scalars: {p_end} {pstd}{cmd:r(Q)}{space 6}Cochran's Q statistic {p_end} {pstd}{cmd:r(df)}{space 5}Degrees of freedom for Cochran's Q statistic {p_end} {pstd}{cmd:r(p)}{space 6}P-value for Cochran's Q statistic {p_end} {pstd}{cmd:r(tau)}{space 4}Heterogeneity standard deviation {p_end} {pstd}{cmd:r(rho)}{space 4}Reliability {p_end} {pstd}Matrices: {p_end} {pstd}If -by- is used, instead of scalars, statistics are saved into vectors. Also, a local macro {cmd:r(levels)} holds the -by- levels for reference. {pstd}{cmd:r(Q)}{space 6} Vector of Cochran's Q statistics, by group {p_end} {pstd}{cmd:r(df)}{space 5} Vector of degrees of freedom for Cochran's Q statistic, by group {p_end} {pstd}{cmd:r(p)}{space 6} Vector of p-values for Cochran's Q statistic, by group {p_end} {pstd}{cmd:r(tau)}{space 4} Vector of heterogeneity standard deviation, by group {p_end} {pstd}{cmd:r(rho)}{space 4} Vector of reliabilities, by group {p_end} {title:Authors} {pstd}Laura Bellows, Duke University, USA{break} laura.bellows@duke.edu {pstd}Paul von Hippel, University of Texas at Austin, USA{break} paulvonhippel.utaustin@gmail.com {title:References} {pstd}von Hippel, Paul T., and Laura Bellows (2018). "How Much Does Teacher Quality Vary Across Teacher Preparation Programs? Reanalyses from 6 States." Economics of Education Review, in press. Also available as SSRN Working Paper, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2990498. {pstd}Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558. https://doi.org/10.1002/sim.1186 {title:Also see} {pstd}von Hippel, Paul T., Laura Bellows, Cynthia Osborne, Jane Lincove, and Nicholas Mills (2016). "Teacher Quality Differences Between Teacher Preparation Programs: How Big? How Reliable? Which Programs Are Different?" {it}Economics of Education Review 53, {sf}31-45. doi:10.1016/j.econedurev.2016.05.002