help for pstest

Covariate imbalance testing and graphing

Main syntax:

pstest
[varlist]
[if exp]
[in range]
[,
both
raw
treated(varname)
mweight(varname)
support(varname)
rubin
notable
label
onlysig
dist
atu
graph
hist
scatter
graph_options ]

where varlist may contain factor variables; see fvvarlist.

Additionally, pstest allows closer inspection of the extent of balancing of individual continuous covariates using densities or box plots:

pstest
varname
[if exp]
[in range],
density|box
[
both
raw
treated(varname)
mweight(varname)
support(varname)
outlier
title(string)
saving(filename[, replace])
atu
]

where varname is a continuous variable.

Description - Main syntax

pstest calculates and optionally graphs several measures of the extent of balancing of the variables in varlist
between two groups (if varlist is not specified, pstest will look for the variables that were
specified in the latest call of psmatch2 or of pstest). In particular it can be used to gauge comparability in terms of varlist between:

1. Two matched samples (the default).

pstest can be called directly after psmatch2, or
it can be fed matching weights via option mweight to assess the extent of balancing achieved
on the two matched samples. A particularly useful way to use pstest is in search of a matching method and set
of matching parameters that achieves good balancing; psmatch2 can be called repeatedly prefixed by quietly
and the extent of corresponding balancing can each time be displayed by calling pstest.

2. Any two samples (option raw).

pstest can be called to assess the comparability of any two groups. This may be
before performing matching, or completely unrelated to matching purposes. (The groups are in any case
referred to as Treated and Controls, but they could be males and females, employed and non-employed etc.).

3. Two samples before and after having performed matching (option both).

In this case pstest compares the extent of balancing between the two samples before
and after having performed matching.

For each variable in varlist it calculates the following indicators (before
and after matching if option both is specified):

(a) t-tests for equality of means in the two samples. T-tests are based on a regression of the variable on a treatment indicator.
Before matching or on raw samples this is an unweighted regression on the whole sample, after matching the regression is weighted using the
matching weight variable _weight or user-given weight variable in mweight and based on the on-support sample;

(b) the standardised percentage bias. If option both is specified, the standardised percentage bias is shown before and after matching,
together with the achieved percentage reduction in abs(bias).
The standardised % bias is the % difference of the sample means in the treated and non-treated (full or
matched) sub-samples as a percentage of the square root of the average of the
sample variances in the treated and non-treated groups (formulae from Rosenbaum and Rubin, 1985).

(c) the variance ratio (for continuous covariates) of treated over non-treated. This ratio should equal 1 if there is perfect balance. An asterisk
is displayed for variables that have variance ratios that exceed the 2.5th and 97.5th percentiles of
the F-distribution with (number of [matched] treated minus 1) and (number of [matched] treated minus 1) degrees of freedom.
These F-percentiles only offer a rough guide and are reported at the bottom of the table (see Austin, 2009).
Alternatively, if option rubin is specified, the ratio of the variance of the residuals orthogonal to the linear index of the
propensity score in the treated group over the non-treated group is shown for each covariate. One asterisk is displayed for variables "of concern" -
ratio in [0.5, 0.8) or (1.25, 2]; two asterisks are displayed for "bad" variables - ratio <0.5 or >2 (see Rubin, 2001).

It also calculates the following overall measures of covariate imbalance, where if option both is specified,
the indicators are shown both before and after matching (note that these measures are only meaningful if varlist contains the full
set of covariates of interest):

(a) Pseudo R2 from probit estimation of the conditional treatment probability (propensity score)
on all the variables in varlist on raw samples, matched samples (default) or both before and
after matching. Also displayed are the corresponding P-values of the likelihood-ratio test of the joint insignificance of all the regressors (before and after matching
if option both is specified);

(b) the mean and median bias as summary indicators of the distribution of the abs(bias);

(c) the percentage of continuous variables that have variance ratios that exceed the 2.5th and 97.5th percentiles of the F-distribution, or,
if option rubin is specified, the percentage of all covariates orthogonal to the propensity score with the specified
variance ratios (% of concern and % bad, with % good being the complement to 100);

(d) Rubins' B (the absolute standardized difference of the means of the linear index of the propensity score in the treated and
(matched) non-treated group) and Rubin's R (the ratio of treated to (matched) non-treated variances of the propensity score index).
Rubin (2001) recommends that B be less than 25 and that R be between 0.5 and 2 for the samples to be considered sufficiently balanced.
An asterisk is displayed next to B and R values that fall outside those limits.

Optionally pstest graphs the extent of covariate imbalance in terms of standardised
percentage differences using dot charts (option graph) or histograms (option hist).	Alternatively, option scatter
draws a scatterplot of the standardised differences vs Rubin's residual variance ratios, offering an at-a-glance picture of covariate
imbalance in terms of these two indicators.

One only need type pstest[, both] directly after psmatch2 to inspect the extent of covariate balancing
in matched samples if psmatch2 has been called with a varlist.

If option both is specified, pstest returns the following diagnostics of
covariate balancing before and after matching: r(meanbiasbef) and r(meanbiasaft) the mean absolute standardised bias;
r(medbiasbef) and r(medbiasaft) the median absolute standardised bias;
r(r2bef) and r(r2aft) the pseudo R2 from probit estimation;
r(chiprobbef) and r(chiprobaft) the P-value of the likelihood-ratio test;	r(Bbef) and r(Baft) Rubin's B; and
r(Rbef) and r(Raft) Rubin's R. If the two groups
are compared only once (matched samples as default or two unmatched samples if option raw is specified),
pstest returns r(meanbias), r(medbias), r(r2), r(chiprob), r(B) and r(R).
pstest always returns in r(exog) the names of the variables for which it has tested
the extent of balancing.

Description - Syntax to assess balancing of individual continuous covariates

Calling pstest varname, density|box displays the density or box plot of the specified
continous variable varname.

As in the main syntax, pstest can be used this way to inspect the density/box plot of a variable for any two groups (option raw),
for matched samples (default) and both before and after matching (option both).

This calling of pstest does not return any indicator (and indeed clears any indicator returned in r() left behind
from the main syntax).

Important notes

pstest by default considers balancing for the ATT (Average Treatment Effect on the Treated),
where the treated are the reference group. If called after
psmatch2, ate one can specify the option atu to consider balancing for the ATU
(Average Treatment Effect on the Untreated), where the non-treated are the reference group.

Spline matching as in psmatch2, spline as well as the default (epanechnikov) local
linear regression matching as in psmatch2, llr first smooth the outcome and then perform
nearest neighbor matching. pstest does not make sense in these cases since
more non-treated are used to calculate the counterfactual outcome than the nearest neighbor only.

Detailed Syntax

Matched samples:

pstest
[varlist]
[if exp]
[in range]
[,
treated(varname)
mweight(varname)
support(varname)
rubin
notable
dist
label
onlysig
atu
graph
hist
scatter
graph_options ]

pstest
varname
[if exp]
[in range],
density|box
[
treated(varname)
mweight(varname)
support(varname)
outlier
title(string)
saving(filename[, replace])
atu
]

Raw samples:

pstest
[varlist]
[if exp]
[in range],
raw
treated(varname)
[
rubin
notable
dist
label
onlysig
graph
hist
scatter
graph_options ]

pstest
varname
[if exp]
[in range],
density|box
raw
treated(varname)
[
outlier
title(string)
saving(filename[, replace])
]

Before and after matching:

pstest
[varlist]
[if exp]
[in range],
both
[
treated(varname)
mweight(varname)
support(varname)
rubin
notable
dist
label
atu
graph
hist
scatter
graph_options ]

pstest
varname
[if exp]
[in range],
density|box
both
[
treated(varname)
mweight(varname)
support(varname)
outlier
title(string)
saving(filename[, replace])
atu
]

Options

both Requires comparability to be assessed both before and after matching.
Default is only after matching.

raw Requires comparability to be assessed between any two (unweighted) groups.
This can be before wishing to perform matching, but also unrelated to matching purposes, e.g.
to quickly assess how randomisation has worked.

treated(varname) Treatment (or group) indicator (0/1).
If option raw is not specified, default is _treated left behind from the latest psmatch2 call.

mweight(varname) Weight of matches.
If option raw is not specified, default is _weight left behind from the latest psmatch2 call.

support(varname) Common support indicator (0/1).
If option raw is not specified, default is _support left behind from the latest psmatch2 call.

rubin Display Rubin's (2001) ratio of the variance of the covariates orthogonal to the propensity score
(instead of the standard variance ratios) in the variable-by-variable table and the percentage of all covariates
orthogonal to the propensity score (instead of the percentage of continuous covariates) with the specified variance ratios in the overall table.

notable Do not display the table with the individual covariate imbalance
indicators (standardised percentage bias, t-tests, if option both is specified
achieved percentage reduction in absolute bias) and variance ratios for each variable in varlist.

label Display variable labels instead of variable names in the variable-by-variable
table.

onlysig In the variable-by-variable table only display those variables
which are significantly unbalanced (p<=0.10). This option is ignored if option both is specified.

dist Display the distribution summary of the absolute standardised percentage
bias across all variables in varlist.

graph Display a graphical summary of covariate imbalance via a dot chart, showing the
standardised percentage bias for each covariate. If option both is specified, information
before and after matching is displayed in the same dot chart. If more than 30 covariates are specified, they are not labelled.

hist Display a graphical summary of covariate imbalance via a histogram,
showing the distribution of the standardised percentage bias across covariates. If option both is specified,
imbalance before and after matching is displayed in two histograms forced to have the same x-axis. Recommended for a large number of covariates.

scatter Display a graphical summary of covariate imbalance via a scatter plot of
standardised percentage bias vs residual variance ratio. If option both is specified,
imbalance before and after matching is displayed in two scatterplots forced to have the same x-axis.
Note that specifying scatter implicitly specifies rubin, so that information on Rubin's (2001) ratio of the variance of the
covariates orthogonal to the propensity score is displayed in the tables.

graph_options Additional options can be specified for the relevant graph type
(dot graph or histogram). Useful examples are saving(filename[, replace]); yscale(range(numlist)),
ylabel(numlist)) or legend(off) for dot graphs; and bin(#) for histograms.

atu After psmatch2, ate one can specify this option to consider balancing when the untreated make up the reference group.

Options specific to the second syntax:

density or box One of these options has to be specified in this case. If density is
specified, the kernel density of varname is estimated and plotted, if box is specified, the box plot of varname is displayed.
A box plot displays the following, from top to bottom: the top outside values (if option outlier is specified),
the upper adjacent value, the 75th percentile, the median, the 25th percentile, the lower adjacent value
and (if option outlier is specified) the bottom outside values of the variable varname.

outlier In a box plot, display the outside values.

title(string) For either density or box plots, specify this option
to override the default title (given by the label of varname, and in the absence of a label, the name of varname).
Do NOT contain the title in quotes, e.g. a title could be: title(Kernel density estimate of real wages).

saving(filename[, replace]) To save the graph produced by density or box.

Examples

. pstest age gender foreign exper, t(training) mw(_weight) onlysig graph
. pstest foreign##c.age married##c.exper if district==1, raw t(male) label scatter

. psmatch2 treated age gender foreign exper, outcome(wage) ate
. pstest
. pstest, both
. pstest, both atu

. pstest age, box out raw t(treated) saving(age_raw, replace)
. pstest earn if earn<40000, density both title(Balancing of earnings before and after matching)

Also see

The commands psmatch2, psgraph.

References

Austin, P.C. (2009), "Balance Diagnostics for Comparing the Distribution of Baseline Covariates Between Treatment Groups in Propensity Score Matched Samples." Statistics in Medicine 28(25), 3083-3107.

Rosenbaum, P.R. and Rubin, D.B. (1985), "Constructing a Control Group Using Multivariate Matched Sampling Methods that Incorporate the Propensity Score", The American Statistician 39(1), 33-38.

Rubin, D.B. (2001), "Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation", Health Services & Outcomes Research Methodology 2, 169-188.

Authors

Edwin Leuven, University of Oslo. If you observe any problems mailto:e.leuven@gmail.com.

Barbara Sianesi, Institute for Fiscal Studies, London, UK.