Help for xtoos_bin_i

Description

xtoos_bin_i evaluates the out-of-sample prediction performance of a specified panel data model with a binary dependent variable in a cross-individual dimension.

The procedure excludes a group of individuals (e.g.countries) defined by the user from the estimation sample (including all their observations throughout time).
Then for each remaining subsample it fits the specified model and uses the resulting parameters to predict the probability of ocurrence of the dependent variable in the unused individuals (out-of-sample).

The procedure evaluates the prediction performance based on the area under the receiver operator characteristic (ROC) statistic evaluated in both the training sample (labeled as _in) and the out-of-sample (labeld as _oos).
The procedure reports both the estimated ROC and also its standard error (SE).

xtoos_bin_i allows to choose different estimation methods (e.g.logit, probit, xtprobit) and could also be used in a cross-section dataset only.

The individuals excluded (learning sample) could be:

i) Random subsamples of size n; if the whole sample contains N individuals, then N/n subsamples without repeated individuals are extracted and evaluated. Moreover, the sampling process could be repeated r times.

ii) an ordered partition of the sample in subsamples of size k; if the whole sample contains N individuals, then N/k ordered subsamples are formed and evaluated.

iii) a particular individual or a particular group (e.g.a country or a region).

Proceedings Spain Stata Conference 2019

https://www.stata.com/meeting/spain19/slides/Spain19_Ugarte-Ruiz.pdf

Syntax

xtoos_bin_i depvar [indepvars] [if], *ous(integer) [rsmpl(integer)] *ksmpl(integer)
[mprob(string)] [evalopt(varname)] [met] [fe]
[model_options]

Options

*ous() It is always required. Specifies the size n of each random subsample (i.e. the number of individuals) that is excluded from each (in)sample estimation, and for which the prediction performance is evaluated.

rsmpl() Specifies the number r of times the random sampling process is repeated. The default is (1). If this parameter is defined as zero, the procedure will not perform any random sampling.

*ksmpl() It is always required. Specifies the size k of each partition or (non-random) subsample, i.e. the number of individuals in each partition, and for which the prediction performance is evaluated. The size k automatically determines the number of subsamples. If this parameter is defined as zero, the procedure will not perform any partition.

mprob() Specifies the method of estimating the probability of a positive outcome that depends on the estimation method used (e.g. prob, pu0, pc1).

evalopt() Specifies a particular individual or group of individuals to be excluded from the estimation (in)sample and for which the prediction performance is evaluated. The particular individual or group must be defined by a dummy variable equal to one for those individuals.

met() Specifies the estimation method. The default is xtlogit.

fe Estimates using Fixed-Effects (within) estimator.

model_options Specifies any other estimation options specific to the method used and not defined elsewhere.

Example 1. Defining the evaluation sample (out-of-sample individuals)

Use of xtoos_bin_i to evaluate the prediction perfomance based on ROC for 3 random subsamples of 1500 individuals (rsmpl() and ous())
and ordered subsamples of also 1500 individuals (ksmpl())

. webuse union, clear

. xtset idcode year

. xtoos_bin_i union age grade i.not_smsa south##c.year, k(1500) o(1500) r(3) mprob(pr)

Example 2. Fixed Effects option and probability method

Use of xtoos_bin_i using Fixed-Effects (within) estimator and the estimation option pc1 for the predicted probability

. xtoos_bin_i union age grade i.not_smsa south##c.year, k(1500) o(1500) r(3) mprob(pc1) fe

Example 3. Changing the estimation method

Use of xtoos_bin_i using as estimation method the command xtprobit

. xtoos_bin_i union age grade i.not_smsa south##c.year, k(1500) o(1500) r(3) mprob(pr) met(xtprobit)

Example 4. Restricting the evaluation to a group of individuals

Use of xtoos_bin_i to evaluate the prediction perfomance based on ROC, but restricting the evaluation to the first 50 individuals

. gen id1to50=idcode<=50

. xtoos_bin_i union age grade i.not_smsa south##c.year, k(1500) o(1500) r(3) mprob(pr) evalopt(id1to50)

Stored results

xtoos_bin_i displays one table for each possible way of generating an learning sample and of evaluating the model's out-of-sample performance, i.e. either random-sampling, ordered partition or particular individuals.

xtoos_bin_i stores the following in r():

Matrices

r(random) evaluation results after random-sampling

r(ordered) evaluation results for ordered partition

r(specific) evaluation results for particular individuals

Author

Alfonso Ugarte-Ruiz
alfonso.ugarte@bbva.com