Title
isa -- Imbens' (2003) sensitivity analysis
Syntax
isa depvar treatmentvar [indepvars] [if] [in] [weight] , tau(real) [options]
options Description ------------------------------------------------------------------------- Model * tau(real) the target size of the coefficient of the treatment variable. observation(int) the number of observations; default is observation(20) resolution(int) the number of iterations for each fixed value of alpha; default is resolution(20) minalpha(real) the minimum value of alpha in the iterations; default is minalpha(0) maxalpha(real) the maximum value of alpha in the iterations; default is maxalpha(10) mindelta(real) the minimum value of delta in the iterations; default is mindelta(0) maxdelta(real) the maximum value of delta in the iterations; default is maxdelta(5) nodots suppresses display of iteration dots.
SE/Robust vce(vcetype) vcetype may be oim, robust, cluster clustvar, or opg
Graph nograph suppresses the figure noplot suppresses the plots of the partial R-sq of each control variable in the figure nplots(int) specifies the number of control variables plotted on the graph; default is nplots(5). The first n variables in indepvars are selected.
Advanced precision(real) the percentage error from tau(real) within which iteration ends; default is precision(0.1) sminalpha(real) the minimum value of alpha in the 2nd run of the iterations; default is aminalpha(0) smaxalpha(real) the maximum value of alpha in the 2nd run of the iterations; default is amaxalpha(10) smindelta(real) the minimum value of delta in the 2nd run of the iterations; default is amindelta(0) smaxdelta(real) the maximum value of delta in the 2nd run of the iterations; default is amaxdelta(5) ml_iterate(int) specifies the number of iterations in each MLE; default is ml_iterate(5) skiprangecheck skips the check of initial values of minalpha, minalpha, minalpha and minalpha. showrangecheck shows the results of the rangecheck performed in Step 2. quick directs - isa- to switch the role of alpha and delta at an earlier point than default. tstat(real) the target size of the t-value of the treatment variable.
------------------------------------------------------------------------- * tau(real) is required. * The value of tau(real) must be positive and smaller than the original treatment effect. At least one indepvars is required. treatmentvar must be binary. aweights and fweights are allowed; see weight. aweight in -logit- is no longer supported in the current version of Stata. Thus, -isa- uses -logit- in version 9.
Description
-isa- produces a figure for the sensitivity analysis developed by Imbens (2003). Observational studies cannot control for the bias due to the omission of unobservables. The sensitivity analysis provides a benchmark about how strong assumption about unobservables researchers need to make to maintain the causal interpretation.
Specifically, Imbens (2003) set up the situation in which researchers can manipulate the magnitude of the bias by changing the coefficient of an unobservable on a treatment variable (alpha) and that on an outcome variable (delta). Thus, there exist a number of pairs of alpha and delta that change the treatment effect by a given size. In principle, Imbens' (2003) sensitivity analysis presents the contour of an equal bias having alpha in the horizontal axis and delta in the vertical axis.
-isa- automatically maximizes the likelihood function developed by Imbens (2003), find such pairs with an efficient algorithm, and draw a contour by connecting these pairs. How to interpret the resulting figure is explained below in Example.
Options
+-------+ ----+ Model +------------------------------------------------------------
tau(real) specifies the target size of the coefficient of the treatment variable. For example, if a researcher finds the treatment effect of 1.7 and wants to know the strength of the confounding by an unobservable that halves the coefficient, s/he should set tau(0.85). The contour in the figure represents the set of partial effects of an unobservable that changes the coefficient to 0.85.
observation(int) specifies the number of points at which the value of alpha in the horizontal axis is fixed. The values of alpha are automatically selected based on the number of observation and the range of alpha. In the 2nd run, the roles of alpha and delta are switched. In general, a larger value of observation(int) results in a better figure at the cost of computation time.
resolution(int) specifies the number of iterations in finding the value of delta for each fized value of alpha. The algorithm searches delta by splitting the range of delta into half in each iteration. Thus, the resolution specifies how fine estimate about delta a researcher wants to obtain. For example, when the range of delta is [0,8], resolution(1) only tells us whether the value of delta lies in [0,4) or [4,8] while resolution(3) provides the value of delta at one-unit precision, say [3,4) or [4,5]. Default is resolution(20). In the 2nd run, the roles of alpha and delta are switched. In general, a larger value of resolution(int) results in a more accurate figure at the cost of computation time.
minalpha(real) and maxalpha(real) specify the range of alpha within which the values of alpha are searched. In Imbens (2003), alpha denotes the coefficient of unobservable in the treatment assignment equation and is represented by the horizontal axis. In the first run, -isa- fixes the value of alpha at observation(int) different points and, for each value of alpha, finds delta that changes the treatment effect to a given amount by changing its value. In the second run, alpha plays the same role as delta in the first run.
mindelta(real) and maxdelta(real) specify the range of delta within which the values of delta are searched. In Imbens (2003), delta denotes the coefficient of unobservable in the outcome equation and is represented by the vertical axis. In the first run, for each value of alpha, -isa- finds delta that changes the treatment effect to a given amount by changing its value. The value of delta is searched until the range of delta is halved by resolution(int) times, or the updated treatment effect is close enough to the target value. In the second run, alpha plays the same role as delta in the first run.
+-----------+ ----+ SE/Robust +--------------------------------------------------------
vce(vcetype) "specifies the type of standard error reported, which includes types that are derived from asymptotic theory, that are robust to some kinds of misspecification that allow for intragroup correlation; see [R] vce_option." (*cited from Stata manual.)
+----------+ ----+ Advanced +---------------------------------------------------------
precision(real) specifies the percentage error from tau(real) at which iteration ends. The default is precision(0.1), which means that when a researcher sets tau(.314), for each fixed value of alpha(1st run) and delta(2nd run), -isa- stop iteration and move to the next value if the t-statistics of the treatment effect take any value between .313686(99.9% of .314) and .314314(100.1% of .314). There is a trade off between computational time and accuracy, but the default value should be sufficiently small.
sminalpha(real) and smaxalpha(real) specify the range of alpha within which the values of alpha are searched in the second run. In the second run, for each value of delta, -isa- finds alpha that changes the treatment effect to a given amount by changing its value. The value of alpha is searched until the range of alpha is halved by resolution(int) times, or the updated treatment effect is close enough to the target value. Researchers may set these options if they want to narrow down the range of alpha in the second run.
smindelta(real) and smaxdelta(real) specify the range of delta within which the values of delta are searched in the second run. In the second run, -isa- fixes the value of delta at observation(int) different points and, for each value of delta, finds alpha that changes the treatment effect to a given amount by changing its value. Researchers may set these options if they want to increase the range of delta in the second run because needed to set the narrow range for delta in the 1st run.
skiprangecheck skips the range check of initial values of minalpha, minalpha, minalpha and minalpha. This option should be specified if researchers are certain about their specification of the initial values. (e.g., when researchers ran -isa_search- before running -isa-.)
showrangecheck shows the results of the range check performed in Step 2. This option helps users figure out how much change in alpha or delta is necessary for the program to pass the range check. If the dataset is small or highly unbalanced, -isa_search- is better option to specify the values of alpha and delta.
quick directs - isa- to move to the next step when the contour plots are more likely to be found if it switches the role of alpha and delta. Specifically, -isa- moves to the next step if it cannot find delta smaller than 75% of the target value for a given fixed alpha or delta. For small, unbalanced data, it is possible, however, that this option mistakenly skips the rest of iterations due to the finding of irrelevant contour.
tstat(real) specifies the target size of the t-statistics of the treatment variable. Because the standard errors produced by -isa- are not accurate, the users should use -gsa- if they want to draw the bound in terms of test statistics. See Harada (2012) for the detail.
Example
The example below evaluates the effect of union membership on hourly wage assuming that union membership is a treatment variable.
. sysuse nlsw88 . xi i.race . rename _Irace_2 black . rename _Irace_3 other . xi i.industry . rename _Iindustry_5 mnfctr . isa wage union grade mnfctr south black age other married c_city, tau(.314)
If covariates in the model capture all factors that affect the wage and the likelihood of becoming a union member, the treatment effect is unbiased. However, researchers might be worry about the possibility that some unobserved individual predispositions affect the decision of becoming a union membership and the wage. In the original model, union membership increases the hourly wage by .628 , so I set tau(.314), which reduces the treatment effect into half.
In 10 minutes or so, -isa- produces the figure with a convex contour and several plots below the contour. This contour indicates the statistical power that an unobservable needs to have for the treatment effect to become half. For example, an unobservable needs to have the partial R-squares of 0.025 in the outcome equation and 0.1 in the treatment assignment equation to halve the treatment effect. Several plots provide further information about the strength of this assumption. These plots represent the partial R-squares when each of them is omitted from the models. In other words, these plots show the statistical power of each covariate.
For instance, the plots of grade (years of education) and mnfctr (manufacturing sector job) are located near the contour. This indicates that if there is an unobservable of which effects on the outcome variable and the treatment variable are as strong as the effects of grade and mnfctr, the treatment would become half. Whether this assumption is strong or weak depends on researchers substantive knowledge. For example, if a researcher believes that an unobservable that is comparable to education is unlikely to exist, the treatment effect of union membership would be fairly robust to against unobserved confounders.
Tips
1. Do not set maxalpha and maxdelta too large. This increases the probability that -isa- finds the irrelevant contour, which messes up the algorithm. Default values often specify the reasonable range.
2. If -isa- produces oddly-shaped, too-short, or no contour, try -isa_search-. If a contour is almost OK but you want a better contour, you may only need to adjust some options such as observation, resolution, minalpha, maxalpha, mindelta, and maxdelta.
3. After -isa_search-, if setting minalpha, maxalpha, mindelta, and maxdelta according to the result of -isa_search- still does not produce a good contour, set sminalpha, smaxalpha, smindelta, and smaxdelta. However, this does not necessarily mean that setting these options always increases the probability of producing a good contour. Therefore, try both with and without these options.
4. You can manually drop problematic observations that produce the irrelevant contour(s) and reproduce the contour using -isa_graph-.
5. Without setting sminalpha, smaxalpha, smindelta, and smaxdelta, the options "skip obs(200)" may help produce a contour. In this case, you probably need to drop some problematic results manually.
Saved results
-isa- saves the following variables. The graph can be reproduced using these saved variables. The results of _n th iteration are recorded in the _n th row:
isa_tau the updated treatment effect when the coefficients of unobservable are alpha and delta isa_se the standard error of the updated treatment effect isa_t the t-statistics of the updated treatment effect isa_alpha the value of alpha isa_delta the value of delta isa_partial_rsq_y the partial r-square of an unobservable in the outcome equation isa_partial_rsq_t the partial r-square of an unobservable in the treatment assignment equation isa_converge identifier of whether the estimation achieved convergence or not isa_partial_rsq_yx the partial r-square of the k th covariate in the outcome equation isa_partial_rsq_tx the partial r-square of the k th covariate in the treatment assignment equation isa_plotvar the name of the k th covariate
Also see
Help: isa_search and isa_graph
Reference
Here is the link for the quick guide by the author of the program.
If you use this program, please cite:
Imbens, Guido W. 2003. "Sensitivity to Exogeneity Assumptions in Program Evaluation." The American Economic Review 93(2):126-132.
Harada, Masataka "Generalized Sensitivity Analysis." Working paper.
Contact
Please feel free to contact Masataka Harada(masatakaharada@nyu.edu) for