*! version 1.0 14dec2014

help overfitdialog:

Title

overfit -- calculates shrinkage statistics to measure overfitting as well as out- and in-sample predictive bias

Syntax

overfit [, options] : est_command est_arguments [, est_options]

where est_command is any estimation command that models the expectation of a quantitative outcome y, and for which post-estimation command predict can predict the expected value of y.
Refer to the help of the estimation command for the syntax of its arguments (est_arguments) and options (est_options).
Note that overfit neither permits the use of if, in and using conditions nor weights in the estimation command.

Main
predopt(string) specify options to the predict command

Cross-validation
nbgrp specify the number of groups

nbiter specify the number of repetitions

seed initialize the random number generator

splitnorand perform a non-random group assignment

efficient use a more efficient but more computer-intensive method

Display
noiterbar prevent overfit from showing the iteration progress

noresults prevent overfit from displaying the results

hist(string) display and save histograms of the shrinkage statistics

showmod display the estimated model for each group at each iteration

showslopes display the estimated slope statistics at each iteration

Save detailed results
savemod(string) save the model's estimated coefficients for each group at each iteration

savepred(string) save all predictions generated by the estimated model of each group at each iteration

procnb(integer) set the processor number

Description

Command overfit calculates three shrinkage statistics to measure the amount of overfitting generated by an estimated model as well as its out- and in-sample predictive bias as defined in Bilger and Manning (2015).
Note that in the case of nonlinear models, all these shrinkage statistics are calculated on the (untransformed) scale of interest, which is also referred to as the "raw scale" of the outcome.

In addition to quantifying overfitting and predictive biases, the purpose of these three measures is to provide guidance to the analyst when chosing a model specification.
When overfitting is severe, reducing the model's flexibility by decreasing its nonlinearity and number of explanatory variables can potentially be beneficial.
On the other hand, when the model's in-sample predictive bias is severe, using a more flexible nonlinear form and adding explanatory variables can potentially be beneficial.
Both overfitting and in-sample bias adversly affect the out-of-sample predictive (or forecasting) performance of the model,
and it is important to measure the resulting out-of-sample predictive bias of the model when it is used to predict (or forecast) new outcomes.
These shrinkage statistics complement other model selection statistics such as Mallow's Cp and the Akaike information criterion by focusing on predictive calibration
(how good the predictions are on average as opposed to how well individual outcomes are predicted).
As such, shrinkage statistics are often of interest per se, for instance in economics for government budgeting or health insurance risk adjustment where getting the average prediction of a given population right is important.

Specifically, overfit calculates the following quantities:

(1 - delta) measures the out-of-sample shrinkage or expansion (when the quantity is negative) that arises when the estimated model is used to predict new outcomes.
Such shrinkage or expansion is caused by both model mispecification and overfitting.
Note that delta is obtained by regressing the observed outcomes on their out-of-sample predictions.
For instance, if (1 - delta) equals 5%, the deviations above (below) the outcome's average are understimated (overestimated) by out-of-sample predictions by 5%.

(1 - alpha) measures the in-sample shrinkage or expansion (when the quantity is negative) that arises when the estimated model is used to predict outcomes from the estimation sample.
Such shrinkage or expansion is caused by model mispecification in the estimation sample.
Note that alpha is obtained by regressing the observed outcomes on their in-sample predictions.
For instance, if (1 - alpha) equals 5%, the deviations above (below) the outcome's average are understimated (overestimated) by in-sample predictions by 5%.

(1 - gamma) measures shrinkage that results from overfitting alone.
This quantity is notably immune to any shrinkage caused by in-sample misspecification,
and is obtained from the following relation:
(1 - delta) = (1 - alpha) + alpha * (1 - gamma).
For instance, if (1 - gamma) equals 5%, the deviations above (below) the outcome's average are understimated (overestimated) by out-of-sample predictions by 5% due to overfitting alone.

All shrinkage statistics are calculated using repeated k-fold cross-validation and their means (expressed in percentage) are reported along with their standard errors.
More information on the interpretation and measurement of the above quantities is provided in Bilger and Manning (2015).

Options

Main

predopt(string) specifies option(s) string to post-estimation command predict.
The option(s) must ensure that predict calculates the expected value of the outcome on the raw scale, e.g. predopt("mu").

Cross-validation

nbgrp(integer) specifies the number of groups k for the k-fold cross-validation. Default is 10.

nbiter(integer) specifies how many times the repeated k-fold cross-validation has to be repeated. Default is 100.

seed(integer) initializes the random number generator. Default is 1.

splitnorand assignes the observations to the cross-validation groups on a non-random basis by keeping the ordering of the data in memory unchanged.
Observations 1.._N/nbgrp are assigned to group 1, observations _N/nbgrp+1..2*_N/nbgrp are assigned to group 2, and so on.
Note that iterating the process would always yield the same result, which is why nbiter is automatically set at 1 when splitnorand is specified.

efficient requires overfit to make use of all in-sample predictions when estimating the in-sample slope.
This results in a more efficient estimation of (1 - alpha) and (1 - gamma).
This option slows the estimation down and can be intractable as it needs to internally set nobs at _N * nbgrp.
It is recommended to use option efficient only for small samples where efficiency is an important consideration.
The default is to use only one in-sample prediction per observation.

Display

noiterbar prevents overfit from displaying dots to show the iteration progress.

noresults prevents overfit from displaying the estimated shrinkage statistics.

hist(string) displays histograms of the shrinkage statistics and saves them into file string.gph.

showmod displays the estimated model for each group at each iteration.
Note that nbgrp * estimation results will be displayed if this option is specified.

showslopes displays the estimated slope statistics alpha and delta for each iteration.

Save additional results

savemod(string) saves all estimated coefficients of the model into file string.dta.
In the saved file, column iter indicates the iteration number and column estnb the cestimation number for the corresponding iteration, followed by the coefficient estimates.
Note that nbgrp * estimation results will be saved if this option is specified.

savepred(string) saves all predictions along with the observation and group allocation ids, iteration and estimation numbers,
and dependent variable into file string.dta.
Note that _N * nbgrp * nbiter predicted values will be saved if this option is specified.

procnb(integer) add processor number integer in the name of temporary datasets as follows: ___procnb_tempdata*.
By avoiding potential conflicts, this allows safe multiprocessing.
Default value is 1. Only positive integers are allowed.

Stored results

overfit saves the following in r():

Scalars

r(missingvalues) equals 1 if at least one iteration is missing, and 0 otherwise

r(nbcrashes) number of crashes encountered when estimating the model, gamma and alpha

Matrices

r(shrinkage_iter) estimated shrinkage statistics for each iteration.

r(shrinkage_mean) average shrinkage statistics with their standard error in line 2.
In case crashes occur, either during model estimations or during the estimation of delta and alpha,
the average shrinkage statistics are computed using all iterations free of crashes,
and line 3 displays the number of observations taken into account.

r(crashes) shows, for each iteration, how many crashes occurred during the estimation of the model, delta and alpha.

Examples

Import simulated example-dataset

. net install overfit

. net get overfit

Calculate the shrinkage statistics for a log-gamma GLM using the default option values

. glm healthexp gender age logincome shi phi, link(log) family(Gamma)

. overfit: glm healthexp gender age logincome shi phi, link(log) family(Gamma)

In this example, deviations above (below) the mean are underestimated (overestimated) by 6.8% when the estimated model is used to predict (or forecast) new outcomes.

Out-of-sample shrinkage mainly comes from in-sample mispecification (4.8%) as overfitting causes less shrinkage (2.1%).

Example with more flexible age specification and additional covariates

. glm healthexp gender age age2 age3 age4 age5 logincome shi phi diabetes hypertension obesity cholesterol smoker drinker inactive, link(log) family(Gamma)

. overfit: glm healthexp gender age age age2 age3 age4 age5 logincome shi phi diabetes hypertension obesity cholesterol smoker drinker inactive, link(log) family(Gamma)

The result is an increase in out-of-sample predictive bias mostly caused by an increase in overfitting.

Examples with a wrong specification of income with increasing flexibility

. glm healthexp gender age income income2 shi phi, link(log) family(Gamma)

. overfit: glm healthexp gender age income income2 shi phi, link(log) family(Gamma)

. glm healthexp gender age income income2 income3 shi phi, link(log) family(Gamma)

. overfit: glm healthexp gender age income income2 income3 shi phi, link(log) family(Gamma)

. glm healthexp gender age income income2 income3 income4 shi phi, link(log) family(Gamma)

. overfit: glm healthexp gender age income income2 income3 income4 shi phi, link(log) family(Gamma)

. glm healthexp gender age income income2 income3 income4 income5 shi phi, link(log) family(Gamma)

. overfit: glm healthexp gender age income income2 income3 income4 income5 shi phi, link(log) family(Gamma)

One can see in-sample bias decreasing and overfitting increasing as the polynomial degree of income increases.

Among these polynomial specifications of income, the degree 3 polynomial is the best as it has the lowest out-of-sample bias.

Example with an inadequate link function

. glm healthexp gender age logincome shi phi, link(power 0.5) family(Gamma)

. overfit: overfit: glm healthexp gender age logincome shi phi, link(power 0.5) family(Gamma)

In this example, overfitting is not the main problem and a better in-sample specification is warranted.

More precise estimation of the shrinkage statistics

. overfit, nbiter(250) efficient: glm healthexp gender age logincome shi phi, link(log) family(Gamma)

Example of 2-fold cross-validation

. overfit, nbgrp(2): glm healthexp gender age logincome shi phi, link(log) family(Gamma)

Note the larger standard errors.

Save histograms of the shrinkage statistics as well as the estimated models and predictions for each iteration and each group

. overfit, nbiter(50) hist(myhist) savemod(mymodels) savepred(mypredictions): glm healthexp gender age logincome shi phi, link(log) family(Gamma)

Reference

Bilger M. and W.G. Manning, 2015.
Measuring overfitting in nonlinear models: A new method and an application to health expenditures.
Health Economics 24(1), 75-85.

To Will Manning (1946-2014), a truly wonderful human being and outstanding health economist.

Author

Marcel Bilger, Laboratory of Health Econometrics, Signature Program in Health Services and Systems Research, Duke-NUS Graduate Medical School, Singapore.
Email marcel.bilger@duke-nus.edu.sg with "overfit.ado" as subject if you have any question, comment or suggestion regarding this stata user-written command.