Help for xtoos_i

Description

xtoos_i evaluates the out-of-sample prediction performance of a specified panel data model in a cross-individual dimension.

The procedure excludes a group of individuals (e.g.countries) defined by the user from the estimation sample (including all their observations throughout time).
Then for each remaining subsample it fits the specified model and uses the resulting parameters to predict the dependent variable in the unused individuals (out-of-sample).

xtoos_i Although the default estimation method is xtreg, the procedure allows to choose different estimation methods including some dynamic methodologies and could also be used in a cross-section dataset only.

xtoos_i reports the specified model's forecasting performance, both in absolute terms (RMSE) and also relative to an alternative model by means of an U-Theil ratio, which is the ratio
between the RMSE of the specified model and the RMSE of the alternative model. By default, the alternative model is a "naive" prediction in which the mean of all in-sample individuals
at every period t is used as a prediction for the excluded ones. The procedure also allows to use an AR1 model as the alternative model for the comparison.

It also reports several in-sample and out-of-sample statistics of both the specified and the Alternative model. In the summary table, the in-sample statistics are labeled as "_in"
and the out-of-sample are labeled as "_oos". The procedure reports the in-sample R2 and the out-of-sample pseudo-R2 of both the specified model and of the alternative model, as well as the RMSE of those
same cases, plus the U-theil ratio.

The individuals excluded (learning sample) could be:

i) Random subsamples of size n; if the whole sample contains N individuals, then N/n subsamples without repeated individuals are extracted and evaluated. Moreover, the sampling process could be repeated r times.

ii) an ordered partition of the sample in subsamples of size k; if the whole sample contains N individuals, then N/k ordered subsamples are formed and evaluated.

iii) a particular individual or a particular group (e.g.a country or a region).

Additionally, it allows to draw a graph with the model's prediction for a particular individual or a particular group, when those individuals are not included in the estimation sample.

Proceedings Spain Stata Conference 2019

https://www.stata.com/meeting/spain19/slides/Spain19_Ugarte-Ruiz.pdf

Syntax

xtoos_i depvar [indepvars] [if], *ous(integer) *ksmpl(integer)
[rsmpl(integer)]	[evalopt(varname)] [met] [mcomp(string)] [fe]
[dum] [lags(numlist)] [hgraph] [model_options]

Options

*ous() It is always required. Specifies the size n of each random subsample (i.e. the number of individuals) that is excluded from each (in)sample estimation, and for which the prediction performance is evaluated.

rsmpl() Specifies the number r of times the random sampling process is repeated. The default is (1). If this parameter is defined as zero, the procedure will not perform any random sampling.

*ksmpl() It is always required. Specifies the size k of each partition or (non-random) subsample, i.e. the number of individuals in each partition, and for which the prediction performance is evaluated.
The size k automatically determines the number of subsamples. If this parameter is defined as zero, the procedure will not perform any partition.

evalopt() Specifies a particular individual or group of individuals to be excluded from the estimation (in)sample and for which the prediction performance is evaluated. The particular individual or group must be defined by a dummy variable equal to one for those individuals.

cdate() Specifies the date at which the evaluation performance ends, i.e. the current date or the date at which the sample ends.
It allows Yearly, Quarterly and Monthly dates, or integer numbers, only. Quarterly and Monthly dates should be written as 1990q1 or 1990m1.

met() Specifies the estimation method. The default is xtreg.

mcomp() Specifies the estimation method for the AR1 model used as a comparison and for estimating the U-Theil, when an AR1 is preferred to a "naive" prediction.
The AR1 could be estimated by dynamic panel methods such as xtabond or xtdpdsys, or simply by xtreg or reg.

fe Estimates using Fixed-Effects (within) estimator.

dum Estimates including one dummy variable per individual in the specification.

lags() Specifies the number of lags of the dependent variable that must be included the specification, the default is zero lags.
This option should be used when using a dynamic panel methods such as xtabond or xtdpdsys. If other estimation method is being used,
the command xtoos_t also allows to write down the desired lagged dependent variables terms simply as other explanatory variables.

hgraph Generates a graph with the model's prediction for a particular individual or a particular group, when those individuals are not included in the estimation sample.
The graph displays the actual dependent variable together with the model's prediction for each individual and the alternative (comparison) prediction, which by default
corresponds to the in-sample mean at every period. This option must be run together with the option evalopt().

model_options Specifies any other estimation options specific to the method used and not defined elsewhere.

Example 1. Defining the evaluation sample (out-of-sample individuals)

Use of xtoos_i to evaluate the prediction performance for 20 random subsamples of 40 individuals (rsmpl() and ous())
and ordered subsamples of also 40 individuals (ksmpl())

. webuse abdata, clear

. xtoos_i n w l.w k l.k ys l.ys, ous(40) rsmpl(20) ksmpl(40)

Example 2. Defining lags of the dependent variable

Use of xtoos_i including lags of the dependent variable in the specification. The following three specifications are equivalent:

. xtoos_i n w l.w k l.k ys l.ys, ous(40) rsmpl(20) ksmpl(40) lags(3)

{cmd:. xtoos_i l(0/3).n w l.w k l.k ys l.ys, ous(40) rsmpl(20) ksmpl(40)

. xtoos_i n l.n l2.n l3.n w l.w k l.k ys l.ys, ous(40) rsmpl(20) ksmpl(40) lags(3)

Example 3. Using a dynamic model method

Use of xtoos_i using a dynamic model method, either xtabond or xtdpdsys.
In this case, the default specification includes one lag of the dependent variable

. xtoos_i n w l.w k l.k ys l.ys, ous(40) rsmpl(20) ksmpl(40) met(xtabond)

. xtoos_i n w l.w k l.k ys l.ys, ous(40) rsmpl(20) ksmpl(40) met(xtdpdsys) lags(2)

Example 4. Defining a specific group of individuals to be left out of the sample

Use of xtoos_i evaluating the prediction performance of the first 6 individuals

. gen id1to6=id<=6

. xtoos_i n w l.w k l.k ys l.ys, ous(40) rsmpl(20) ksmpl(40) evalopt(id1to6)

Example 5. Changing the estimation method

Use of xtoos_i using as estimation method the command xtregar

. xtoos_i n w l.w k l.k ys l.ys, ous(40) rsmpl(20) ksmpl(40) met(xtregar)

Example 6. Changing the comparison method

Use of xtoos_i using OLS (reg) as the estimation method for the AR(1) comparison model

. xtoos_i n w l.w k l.k ys l.ys, ous(40) rsmpl(20) ksmpl(40) mcomp(reg)

Example 7. Fixed Effects option

Use of xtoos_i using Fixed-Effects (within) estimator (the estimated individual components cannot be included in the prediction of the out-of-sample individuals)

. xtoos_i n w l.w k l.k ys l.ys, ous(40) rsmpl(20) ksmpl(40) fe

Example 8. Use of dummy variables

Use of xtoos_i using dummy variables per individual (the estimated dummy variable coefficients cannot be included in the prediction of the out-of-sample individuals)

. xtoos_i n w l.w k l.k ys l.ys, ous(40) rsmpl(20) ksmpl(40) dum

Example 9. Drawing a graph with the out-of-sample predictions for a group of individuals

Use of xtoos_i to evaluate the prediction performance restricting the evaluation only to first 6 individuals, while drawing a graph with the prediction
for each one of those 6 individuals

. xtoos_i n w l.w k l.k ys l.ys, ous(40) rsmpl(0) ksmpl(40) evalopt(id1to6) hgraph

Stored results

xtoos_i displays one table for each possible way of generating an learning sample and of evaluating the model's out-of-sample performance, i.e. either random-sampling, ordered partition or particular individuals.

xtoos_i stores the following in r():

Matrices

r(random) evaluation results after random-sampling

r(ordered) evaluation results for ordered partition

r(specific) evaluation results for particular individuals

Author

Alfonso Ugarte-Ruiz
alfonso.ugarte@bbva.com