help opl_overlap

Title

opl_overlap -- Assessing overlap between train and new data

Syntax

opl_overlap
,
[gr_save_ps(filename) gr_save_roc(filename)]

Description

opl_overlap is a post–estimation command of the Optimal Policy Learning (OPL) toolbox.
It is designed to assess the overlap between the covariate distributions of the
training data (data used to learn the optimal policy) and the new data
(data on which the policy is meant to be applied).

The command must be run after make_cate, which sets up the
estimation environment and stores in e() the covariate list used for
policy learning.

The procedure implemented by opl_overlap is as follows:

1. It fits a logistic regression of a binary variable taking value 0 for
observations in the training and 1 for this in the new dataset on the covariates in
e(xvars) returned by make_cate. This model predicts the probability that an observation belongs to the new data rather than to the training data (reverse
propensity score).

2. It predicts the fitted probabilities, i.e. the generated variable _ps_var, and compares their distributions across training and new data by means of
kernel density estimates.

3. It computes the ROC curve and the Area Under the Curve (AUC) using
lroc. The AUC summarizes the degree of separability between
training and new data in terms of the covariate distribution.

If the two datasets have very similar covariate distributions, the reverse propensity score cannot distinguish well between them and the AUC is close to 0.5. Conversely, if the two datasets differ substantially, the AUC moves towards 1, signaling poor overlap and raising concerns about the external validity of the learned policy.

The command produces two graphs:

1) a graph comparing the kernel density of _ps_var in the training
data versus the new data;

2) the ROC curve associated with the logistic model predicting
group.

Internally, opl_overlap generates the following objects:

– variable _ps_var: reverse propensity score (probability of belonging
to the new data);

– graph _GR_PS_: kernel density comparison of _ps_var by group;

– graph _GR_ROC_: ROC curve from lroc.

For safety, the command checks in advance that variable _ps_var and
graphs _GR_PS_ and _GR_ROC_ do not already exist in memory.
If any of them is found, the command throws a warning and exits without
overwriting the existing objects.

Options

options

gr_save_ps(filename) specifies the file where the kernel density comparison graph of the reverse propensity score is to be saved. The in-memory graph name is _GR_PS_.

gr_save_roc(filename) specifies the file where the ROC curve graph is to be saved. The in-memory graph name is _GR_ROC_.

Output and interpretation

opl_overlap prints on the screen a short qualitative assessment of
the overlap between training and new data based on the AUC:

– AUC ≤ 0.60: overlap is classified as "Very Good".

– 0.60 < AUC ≤ 0.70: overlap is classified as "Good".

– 0.70 < AUC ≤ 0.80: overlap is classified as "Moderate".

– 0.80 < AUC ≤ 0.90: overlap is classified as "Poor".

– AUC > 0.90: overlap is classified as "Very poor".

These thresholds provide an empirical rule-of-thumb to judge to what
extent the policy learned on the training data can be reliably
extrapolated to the new data. High AUC values (poor overlap) suggest
that caution is needed when interpreting policy effects on the new
data and may motivate additional robustness checks or re–estimation
on a more comparable sample.

Stored results

opl_overlap stores the following scalars in r():

r(AUC) area under the ROC curve from lroc

r(N) number of observations used to estimate the ROC curve

Remarks

opl_overlap is meant to be used within the OPL workflow after
make_cate has been run. It is particularly useful in applications
where the policy is learned on one dataset (training) and then evaluated
or applied to another dataset (new) – for instance, when using historical
data to design a targeting rule to be implemented on a future population
or on a different region.

A low AUC (close to 0.5) supports the idea that the covariate distribution
in the new data is well represented in the training data, strengthening
the credibility of policy recommendations. A high AUC, instead, warns that
the two datasets are easily separable based on observed covariates, which
may jeopardize the external validity of the learned policy.

Example

Example: Assessing overlap between train and new data

Load initial dataset

sysuse JTRAIN2, clear

Split the original data into a "old" (training) and "new" (testing) dataset

get_train_test, dataname(jtrain) split(0.60 0.40) split_var(svar) rseed(101)

Use the "old" dataset (i.e. policy) for training

use jtrain_train , clear

Set the outcome

global y "re78"

Set the features

global x "re74 re75 age agesq nodegree"

Set the treatment variable

global w "train"

Set the selection variables

global z "age mostrn"

Run "make_cate" and generate training (old policy) and testing (new policy) CATE predictions

make_cate $y $x , treatment($w) type("ra") model("linear") new_cate("my_cate_new") train_cate("my_cate_train") new_data("jtrain_test")

Run the "opl_overlap" command

opl_overlap

Display the AUC

return list

Acknowledgment

The development of this software was supported by FOSSR (Fostering Open Science in Social Science Research), a project funded by the European Union - NextGenerationEU under the NPRR Grant agreement n. MURIR0000008.

Author

Giovanni Cerulli

IRCrES-CNR

Research Institute for Sustainable Economic Growth, National Research Council of Italy

E-mail: giovanni.cerulli@ircres.cnr.it

Also see

Online: make_cate, opl_tb, opl_tb_c, opl_lc, opl_lc_c, opl_dt, opl_dt_c