Title

itsaperm -- Permutation tests for matched multiple group interrupted time series analysis

Syntax

itsaperm depvar [indepvars] [if] [in] [weight],
trperiod(date)
treatid(#)
[ pr(#)
lag(#)
matchvar(varlist)
plot[(twoway_options)]
favors(string # string)
prais
noisily
replace
model_options ]

The panel variable must be declared by using either tsset panelvar timevar
or xtset panelvar timevar. See tsset or xtset.

* trperiod(date) specifies the time period when the intervention begins

* treatid(#) specifies the identifier of the true treated unit

pr(#) specifies the minimum significance level (P-value) for assessing balance

lag(#) specifies the maximum lag to be considered when a newey model is chosen;
the default is lag(0)

matchvar(varlist) specifies the variables used for finding
matches; the default is variables specified in depvar and indepvars

prais fits a prais model; the default model is newey

noisily specifies that estimates be shown in the output window as they are computed; the
default is to show dots

plot[(twoway_options)] produces a forest plot of
the difference-in-differences in trends

favors(string # string) inserts a label saying something about the treatment effect on
either side of zero (strings are separated by the # symbol)

replace replaces variables created by itsaperm if they already exist.

model_options specifies all available options for prais when the prais
option is chosen; otherwise all available options of newey

* trperiod(#) and treatid(#) must be specified.

aweights are allowed when a newey model is specified; see
weight.

Description

itsaperm performs a robustness check in the form of permutation tests to assess whether the treatment
effect (identified in a multiple group interrupted time series analysis with matched sets),
remains plausible under a pseudo-treatment assignment process [Linden 2018b].

itsaperm iteratively casts each unit into the role of “treated”, creating a comparable
control group from other units in the sample using itsamatch, and then evaluates treatment effects using
itsa. The specific treatment effect evaluated is the differences-in-differences in trends (see Remarks
section below). The results of the permutation tests are presented in a forest plot. If statistically significant
“treatment effects” are estimated for pseudo-treated units, then any significant changes in the outcome of the true
treatment unit cannot be attributed to the intervention (see [Linden 2018b] for a detailed discussion).

Options

trperiod(date) specifies the time period (date) when the
intervention begins. The value entered for the time period must be in the same
time unit as the panel time variable specified in tsset timevar; see
tsset. The date should be specified using the respective
pseudofunction (see datetime), such as trperiod(2020)
for a four-digit year, trperiod(2019m11) for quarterly data or
trperiod(08jan2021) for daily data. trperiod() is
required.

treatid(#) specifies the identifier of the true treated
unit. The value entered must be in the same units as the panel variable
specified in tsset panelvar timevar; see tsset.
treatid() is required.

pr(#) specifies the minimum significance level (P-value) for determining
balance on each variable in the varlist. While pr can be set to any value between
0 and 1.0, 0.05 is the usual convention for considering balance. Naturally, higher values will
ensure closer balance, but it comes at a trade-off of losing observations as potential matches.

lag(#) specifies the maximum lag to be considered in the
autocorrelation structure when a newey model is chosen. If the user
specifies lag(0), the default, the output is the same as regress,
vce(robust). An error message will appear if both prais and
lag() are specified, because prais implements an AR(1) model by
design.

matchvar(varlist) specifies the variables that are passed to itsamatch
for finding matched controls. When matchvar is not specified, variables specified in
depvar and indepvars are used for matching (thus ensuring that, at a minimum, the
the treatment unit and its matched controls will be balanced on the preintervention level
and trend of the outcome time series).

prais specifies to fit a prais model. If prais is
not specified, itsa will use newey as the default model.

noisily specifies that the outcome estimates (difference-in-differences in trends) be
shown in the output window as they are computed for each (pseudo)treatment unit. The
default is to show dots representing each unit permuted in the sample.

plot[(twoway_options)] produces
a forest plot of the outcome (difference-in-differences in trends) for all of the permuted
units in the sample (including a blank line for units in which controls could not be found).
Specifying plot without options uses the default graph settings.

favors(string # string) inserts a label saying something about the treatment effect
on either side of zero -- most typically "favors treatment" and "favors control"; strings are separated by the # symbol.

replace replaces the following variables created by itsaperm if they already exist:

id:

Identifier for each unit permuted in the sample. If the panelvar specified in tsset has a label,
it will be assigned to id as well.

idC:

A string variable containing the IDs of controls matched to that unit.

estimate:

The point estimate of the difference-in-differences in trends.

se:

The standard error of the difference-in-differences in trends estimate.

t:

The t-statistic of the difference-in-differences in trends estimate.

p:

The p-value of the difference-in-differences in trends estimate.

lcl:

The lower bound of the 95% confidence interval of the difference-in-differences in trends estimate.

ucl:

The upper bound of the 95% confidence interval of the difference-in-differences in trends estimate.

model_options specify all available options for prais when the
prais option is chosen; otherwise, all available options for
newey other than lag() are specified.

Remarks

The basic regression model for a multiple group ITSA is as follows [Linden and Adams 2011; Linden 2015]:

Y_t = Beta_0 + Beta_1(T) + Beta_2(X_t) + Beta_3(TX_t) +
Beta_4(Z) + Beta_5(ZT) + Beta_6(ZX_t) + Beta_7(ZTX_t)

Beta_0 represents the intercept or starting level of the outcome variable for
the control group,

Beta_1 is the slope or trajectory of the outcome variable
in the control group until the introduction of the intervention,

Beta_2 represents the change in the level of the outcome for the control group that occurs
in the period immediately following the introduction of the intervention (compared with the
counterfactual),

Beta_3 represents the difference between preintervention and
postintervention slopes of the outcome in the control group,

Beta_4 represents the difference in the level (intercept) of the outcome variable between the treatment unit
and controls prior to the intervention,

Beta_5 represents the difference in the slope (trend) of the outcome variable between the treatment unit and
controls prior to the intervention,

Beta_6 indicates the difference between the treatment unit and control groups in the level of the outcome
variable immediately following introduction of the intervention,

Beta_7 represents the difference between the treatment unit and control groups in the slope (trend) of the
outcome variable after initiation of the intervention compared with preintervention (akin to a difference-in-differences
of trends). It is this parameter estimate that itsaperm uses for comparison between
the true treatment unit and all pseudo-treated units.

Examples

Load data and declare the dataset as panel:

. use cigsales, clear

. tsset state year

We generate a single-lag variable of the outcome:

. gen Lcigsale = L1.cigsale

We run permutation tests across all units in the sample data, with the true treated unit being California (3) and the outcome variable "cigsale".
We set the minimum cutoff P-value at 0.40, use cigsale and lagged cigsales for finding matches, specify autocorrelation at lag 1,
and specify the titles on the left and right side of zero to indicate the treatment effect (this replicates data in Figure 3 of [Linden 2018b]).

. itsaperm cigsale, trperiod(1989) treatid(3) matchvar(cigsale Lcigsale) lag(1) pr(0.40) plot favors(Favors treatment # Favors controls)

List control IDs associated with each pseudo-treatment ID (this replicates data in Table 1 of [Linden 2018b]).

. list id idC if id !=.

Same as above but we reverse the yscale and change the presentation of the xlabels. We use replace to replace the variables generated in the previous run.

. itsaperm cigsale, trperiod(1989) treatid(3) matchvar(cigsale Lcigsale) lag(1) pr(0.40) plot(yscale(reverse) xlabel(-6(1)6)) favors(Favors treatment # Favors controls) replace

We add retprice as a matching variable, and set the P-value cutoff to 0.20. We use noisily to see the output as it is produced.

. itsaperm cigsale, trperiod(1989) treatid(3) matchvar(cigsale Lcigsale retprice) lag(1) pr(0.20) plot(yscale(reverse) xlabel(-6(1)6)) favors(Favors treatment # Favors controls) replace noisily

Same as above, but we now estimate the model using the prais option with robust standard errors and change the xlabel specification.

. itsaperm cigsale, trperiod(1989) treatid(3) matchvar(cigsale Lcigsale retprice) pr(0.20) plot(yscale(reverse) xlabel(-10(2)10)) favors(Favors treatment # Favors controls) replace noisily prais vce(robust)

References

Linden, A. 2015.
Conducting interrupted time series analysis for single and multiple group comparisons.
Stata Journal.
15: 480-500.

------. 2017a.
A comprehensive set of postestimation measures to enrich interrupted time-series analysis.
Stata Journal
17: 73-88.

------. 2017b.
Challenges to validity in single-group interrupted time series analysis.
Journal of Evaluation in Clinical Practice.
23: 413-418.

------. 2017c.
Persistent threats to validity in single-group interrupted time series analysis with a crossover design.
Journal of Evaluation in Clinical Practice.
23: 419-425.

------. 2018a.
A matching framework to improve causal inference in interrupted time series analysis.
Journal of Evaluation in Clinical Practice.
24: 408-415.

------. 2018b.
Using permutation tests to enhance causal inference in interrupted time series analysis.
Journal of Evaluation in Clinical Practice.
DOI:10.1111/jep.12899

Linden, A., and J. L. Adams. 2011.
Applying a propensity-score based weighting model to interrupted time
series data: Improving causal inference in program evaluation.
Journal of Evaluation in Clinical Practice
17: 1231-1238.

Citation of itsaperm

itsaperm is not an official Stata command. It is a free contribution
to the research community, like a paper. Please cite it as such:

Linden, Ariel (2018). ITSAPERM: Stata module for conducting permutation tests of matched sets in multiple group interrupted time series analysis

Author

Ariel Linden

Linden Consulting Group, LLC

alinden@lindenconsulting.org

Also see

Help: newey, prais, itsa (if installed), itsamatch (if installed)