help isa-------------------------------------------------------------------------------

Title

isa-- Imbens' (2003) sensitivity analysis

Syntax

isadepvartreatmentvar[indepvars] [if] [in] [weight],tau(real)[options]

optionsDescription ------------------------------------------------------------------------- Model *tau(real)the target size of the coefficient of the treatment variable.observation(int)the number of observations; default isobservation(20)resolution(int)the number of iterations for each fixed value of alpha; default isresolution(20)minalpha(real)the minimum value of alpha in the iterations; default isminalpha(0)maxalpha(real)the maximum value of alpha in the iterations; default ismaxalpha(10)mindelta(real)the minimum value of delta in the iterations; default ismindelta(0)maxdelta(real)the maximum value of delta in the iterations; default ismaxdelta(5)nodotssuppresses display of iteration dots.SE/Robust

vce(vcetype)vcetypemay beoim,robust,clusterclustvar, oropgGraph

nographsuppresses the figurenoplotsuppresses the plots of the partial R-sq of each control variable in the figurenplots(int)specifies the number of control variables plotted on the graph; default isnplots(5). The firstnvariables inindepvarsare selected.Advanced

precision(real)the percentage error fromtau(real)within which iteration ends; default isprecision(0.1)sminalpha(real)the minimum value of alpha in the 2nd run of the iterations; default isaminalpha(0)smaxalpha(real)the maximum value of alpha in the 2nd run of the iterations; default isamaxalpha(10)smindelta(real)the minimum value of delta in the 2nd run of the iterations; default isamindelta(0)smaxdelta(real)the maximum value of delta in the 2nd run of the iterations; default isamaxdelta(5)ml_iterate(int)specifies the number of iterations in each MLE; default isml_iterate(5)skiprangecheckskips the check of initial values of minalpha, minalpha, minalpha and minalpha.showrangecheckshows the results of the rangecheck performed in Step 2.quickdirects -isa- to switch the role ofalphaanddeltaat an earlier point than default.tstat(real)the target size of the t-value of the treatment variable.------------------------------------------------------------------------- *

tau(real)is required. * The value oftau(real)must be positive and smaller than the original treatment effect. At least oneindepvarsis required.treatmentvarmust be binary.aweights andfweights are allowed; see weight.aweightin -logit- is no longer supported in the current version of Stata. Thus, -isa- uses -logit- in version 9.

Description-

isa- produces a figure for the sensitivity analysis developed by Imbens (2003). Observational studies cannot control for the bias due to the omission of unobservables. The sensitivity analysis provides a benchmark about how strong assumption about unobservables researchers need to make to maintain the causal interpretation.Specifically, Imbens (2003) set up the situation in which researchers can manipulate the magnitude of the bias by changing the coefficient of an unobservable on a treatment variable (

alpha) and that on an outcome variable (delta). Thus, there exist a number of pairs ofalphaanddeltathat change the treatment effect by a given size. In principle, Imbens' (2003) sensitivity analysis presents the contour of an equal bias havingalphain the horizontal axis anddeltain the vertical axis.-

isa- automatically maximizes the likelihood function developed by Imbens (2003), find such pairs with an efficient algorithm, and draw a contour by connecting these pairs. How to interpret the resulting figure is explained below inExample.

Options+-------+ ----+ Model +------------------------------------------------------------

tau(real)specifies the target size of the coefficient of the treatment variable. For example, if a researcher finds the treatment effect of 1.7 and wants to know the strength of the confounding by an unobservable that halves the coefficient, s/he should settau(0.85). The contour in the figure represents the set of partial effects of an unobservable that changes the coefficient to 0.85.

observation(int)specifies the number of points at which the value ofalphain the horizontal axis is fixed. The values of alpha are automatically selected based on the number ofobservationand the range ofalpha. In the 2nd run, the roles ofalphaanddeltaare switched. In general, a larger value ofobservation(int)results in a better figure at the cost of computation time.

resolution(int)specifies the number of iterations in finding the value ofdeltafor each fized value ofalpha. The algorithm searchesdeltaby splitting the range ofdeltainto half in each iteration. Thus, the resolution specifies how fine estimate aboutdeltaa researcher wants to obtain. For example, when the range ofdeltais [0,8],resolution(1)only tells us whether the value ofdeltalies in [0,4) or [4,8] whileresolution(3)provides the value ofdeltaat one-unit precision, say [3,4) or [4,5]. Default isresolution(20). In the 2nd run, the roles ofalphaanddeltaare switched. In general, a larger value ofresolution(int)results in a more accurate figure at the cost of computation time.

minalpha(real)andmaxalpha(real)specify the range ofalphawithin which the values ofalphaare searched. In Imbens (2003),alphadenotes the coefficient of unobservable in the treatment assignment equation and is represented by the horizontal axis. In the first run, -isa- fixes the value ofalphaatobservation(int)different points and, for each value of alpha, findsdeltathat changes the treatment effect to a given amount by changing its value. In the second run,alphaplays the same role asdeltain the first run.

mindelta(real)andmaxdelta(real)specify the range ofdeltawithin which the values ofdeltaare searched. In Imbens (2003),deltadenotes the coefficient of unobservable in the outcome equation and is represented by the vertical axis. In the first run, for each value of alpha, -isa- findsdeltathat changes the treatment effect to a given amount by changing its value. The value ofdeltais searched until the range ofdeltais halved byresolution(int)times, or the updated treatment effect is close enough to the target value. In the second run,alphaplays the same role asdeltain the first run.+-----------+ ----+ SE/Robust +--------------------------------------------------------

vce(vcetype)"specifies the type of standard error reported, which includes types that are derived from asymptotic theory, that are robust to some kinds of misspecification that allow for intragroup correlation; see[R] vce_option." (*cited from Stata manual.)

+----------+ ----+ Advanced +---------------------------------------------------------

precision(real)specifies the percentage error fromtau(real)at which iteration ends. The default isprecision(0.1), which means that when a researcher setstau(.314), for each fixed value of alpha(1st run) and delta(2nd run), -isa- stop iteration and move to the next value if the t-statistics of the treatment effect take any value between .313686(99.9% of .314) and .314314(100.1% of .314). There is a trade off between computational time and accuracy, but the default value should be sufficiently small.

sminalpha(real)andsmaxalpha(real)specify the range ofalphawithin which the values ofalphaare searched in the second run. In the second run, for each value of delta, -isa- findsalphathat changes the treatment effect to a given amount by changing its value. The value ofalphais searched until the range ofalphais halved byresolution(int)times, or the updated treatment effect is close enough to the target value. Researchers may set these options if they want to narrow down the range ofalphain the second run.

smindelta(real)andsmaxdelta(real)specify the range ofdeltawithin which the values ofdeltaare searched in the second run. In the second run, -isa- fixes the value ofdeltaatobservation(int)different points and, for each value of delta, findsalphathat changes the treatment effect to a given amount by changing its value. Researchers may set these options if they want to increase the range ofdeltain the second run because needed to set the narrow range fordeltain the 1st run.

skiprangecheckskips the range check of initial values of minalpha, minalpha, minalpha and minalpha. This option should be specified if researchers are certain about their specification of the initial values. (e.g., when researchers ran -isa_search- before running -isa-.)

showrangecheckshows the results of the range check performed in Step 2. This option helps users figure out how much change inalphaordeltais necessary for the program to pass the range check. If the dataset is small or highly unbalanced, -isa_search- is better option to specify the values ofalphaanddelta.

quickdirects -isa- to move to the next step when the contour plots are more likely to be found if it switches the role ofalphaanddelta. Specifically, -isa- moves to the next step if it cannot finddeltasmaller than 75% of the target value for a given fixedalphaordelta. For small, unbalanced data, it is possible, however, that this option mistakenly skips the rest of iterations due to the finding of irrelevant contour.

tstat(real)specifies the target size of the t-statistics of the treatment variable. Because the standard errors produced by -isa- are not accurate, the users should use -gsa- if they want to draw the bound in terms of test statistics. See Harada (2012) for the detail.

ExampleThe example below evaluates the effect of union membership on hourly wage assuming that union membership is a treatment variable.

. sysuse nlsw88 . xi i.race . rename _Irace_2 black . rename _Irace_3 other . xi i.industry . rename _Iindustry_5 mnfctr . isa wage union grade mnfctr south black age other married c_city, tau(.314)

If covariates in the model capture all factors that affect the wage and the likelihood of becoming a union member, the treatment effect is unbiased. However, researchers might be worry about the possibility that some unobserved individual predispositions affect the decision of becoming a union membership and the wage. In the original model, union membership increases the hourly wage by .628 , so I set

tau(.314), which reduces the treatment effect into half.In 10 minutes or so, -

isa- produces the figure with a convex contour and several plots below the contour. This contour indicates the statistical power that an unobservable needs to have for the treatment effect to become half. For example, an unobservable needs to have the partial R-squares of 0.025 in the outcome equation and 0.1 in the treatment assignment equation to halve the treatment effect. Several plots provide further information about the strength of this assumption. These plots represent the partial R-squares when each of them is omitted from the models. In other words, these plots show the statistical power of each covariate.For instance, the plots of

grade(years of education) andmnfctr(manufacturing sector job) are located near the contour. This indicates that if there is an unobservable of which effects on the outcome variable and the treatment variable are as strong as the effects ofgradeandmnfctr, the treatment would become half. Whether this assumption is strong or weak depends on researchers substantive knowledge. For example, if a researcher believes that an unobservable that is comparable to education is unlikely to exist, the treatment effect of union membership would be fairly robust to against unobserved confounders.

Tips1. Do not set

maxalphaandmaxdeltatoo large. This increases the probability that -isa- finds the irrelevant contour, which messes up the algorithm. Default values often specify the reasonable range.2. If -

isa- produces oddly-shaped, too-short, or no contour, try -isa_search-. If a contour is almost OK but you want a better contour, you may only need to adjust some options such asobservation,resolution,minalpha,maxalpha,mindelta, andmaxdelta.3. After -

isa_search-, if settingminalpha,maxalpha,mindelta, andmaxdeltaaccording to the result of -isa_search- still does not produce a good contour, setsminalpha,smaxalpha,smindelta, andsmaxdelta. However, this does not necessarily mean that setting these options always increases the probability of producing a good contour. Therefore, try both with and without these options.4. You can manually drop problematic observations that produce the irrelevant contour(s) and reproduce the contour using -

isa_graph-.5. Without setting

sminalpha,smaxalpha,smindelta, andsmaxdelta, the options "skipobs(200)" may help produce a contour. In this case, you probably need to drop some problematic results manually.

Saved results-

isa- saves the following variables. The graph can be reproduced using these saved variables. The results of_nth iteration are recorded in the_nth row:

isa_tauthe updated treatment effect when the coefficients of unobservable arealphaanddeltaisa_sethe standard error of the updated treatment effectisa_tthe t-statistics of the updated treatment effectisa_alphathe value of alphaisa_deltathe value of deltaisa_partial_rsq_ythe partial r-square of an unobservable in the outcome equationisa_partial_rsq_tthe partial r-square of an unobservable in the treatment assignment equationisa_convergeidentifier of whether the estimation achieved convergence or notisa_partial_rsq_yxthe partial r-square of thekth covariate in the outcome equationisa_partial_rsq_txthe partial r-square of thekth covariate in the treatment assignment equationisa_plotvarthe name of thekth covariate

Also seeHelp: isa_search and isa_graph

ReferenceHere is the link for the quick guide by the author of the program.

If you use this program, please cite:

Imbens, Guido W. 2003. "Sensitivity to Exogeneity Assumptions in Program Evaluation."

The American Economic Review93(2):126-132.Harada, Masataka "Generalized Sensitivity Analysis."

Working paper.

ContactPlease feel free to contact Masataka Harada(masatakaharada@nyu.edu) for