*! version 0.0.10 01mar2024

Computing Model Validation Statistics

Syntax

validateit , metric(string asis)
pstub(string asis) split(varname) [
obs(varname) monitors(string asis)
display kfold(integer) noall loo
name(string asis)]

Required
metric the name of a function from libxv or a user-defined function

pstub variable name stub for predicted values

split name of the variable that identifies the training split(s)

Optional
obs name of the dependent variable; default is obs(`e(depvar)')

monitors zero or more function names from libxv or user-defined functions; default is monitors()

display display results in window; default is off

kfold the number of folds in the training set; default is kfold(1).

noall suppresses prediction on entire training set for K-Fold cases

loo is used only for Leave-One-Out cross-validation

name is used to name the collection storing the results; default is name(xvval).

Description

Options

Required

metric the name of a libxv or user-defined function, with the
function signature described in help libxv used to evaluate the fit
of the model on the held-out data. Only a single metric can be specified. For
user's who may be interested in hyperparameter tuning, this would be the value
that you would optimize with your hyperparameter tuning algorithm. You may pass
optional arguments to metrics that support them by enclosing the arguments in a
transmorphic matrix. Metrics that support optional arguments
will provide guidance on what options are available and how to order/structure
the options for use in the function. For example, this might look like:
metric(mse(("yes", "1"))) or metric(mae((1, 2 \ 3, 4))).

pstub the stub name for the variable(s) that store the predicted values
after calling predictit.

split must contain the name of the variable that stores the test,
validation, and test splits. There will only be a single variable if the splits
were created using splitit.

Optional

obs the name of the variable containing the observed values.

monitors the name of zero or more libxv or user-defined functions,
with the function signature described in help libxv used to
evaluate the fit of the model on the held-out data. These should not be used
when attempting to tune hyper parameters, but can still provide useful
information regarding the model fit characteristics. You may pass optional
arguments to the metrics you list in the monitors option that support them by
enclosing the arguments for each monitor in a
transmorphic matrix. Metrics that support optional arguments
will provide guidance on what options are available and how to order/structure
the options for use in the function. For example, this might look like:
monitors(mse(("yes", "1")) mae((1, 2 \ 3, 4))).

display is an option to display the metric and monitor values in the
results window.

kfold defines the number of K-Folds used for the training set. When the
value is > 1, this will result in metric and monitor computations for each of
the K-Folds. Additionally, it will also compute the metric and monitor values
for the model fitted to all of the training data.

noall is an option to prevent evaluating model performance when using
K-Fold cross-validation and also fitting the model to the full training set.

loo is an option used to alter the underlying logic used to compute the
validation metrics/monitors. Since Leave-One-Out cross-validation is a special
case of K-Fold CV the model fitting and predictions are generated in a manner
consistent with all other K-Fold cases. However, with only a single unit in
each K-Fold in the LOO case computing validation metrics requires different
treatment. In the LOO case the validation metric/monitors are computed using
the predicted and observed values for all of the training set units in aggregate.

name is an option to pass a name to the collection created to store the
results. When validateit is executed, it will initialize a new collection
or replace the existing collection with the same name. If you want to retain
the validation results from multiple executions, pass an argument to this
option. Note: this only affects users using Stata 17 or later.

Custom Metrics and Monitors

Users may define their own validation metrics to be used by validateit.
The name of the function must conform with standard Stata naming conventions
that allow alphanumeric characters and underscores only. Additionally, all
functions are required to use the same function signature:

real scalar metricName(string scalar pred, string scalar obs, string scalar touse, | transmorphic matrix opts)

The first argument passed to your function will be the name of the variable
containing the predicted values. The second argument passed to your function
will be the name of the variable containing the observed outcomes. The third
argument in the signature is a variable that identifies the appropriate data to
use for the metric computation. The last, and only optional argument, is to
allow users greater flexibility in defining custom metrics and to allow our team
to further extend the existing metrics. When passing an argument to the
metric option, you can pass optional arguments to the function in a
transmorphic matrix that you specify. As a hypothetical
example, this could look like:
monitors(mse(("yes", "1")) mae((1, 2 \ 3, 4))). validateit will
handle parsing and passing the matrix to the underlying Mata function for you.
If you wish to allow options for your own custom defined function, remember to
document the option specification clearly and keep things as simple as possible
to minimize user-frustration and problems.

In your function, you can easily define the vectors that will store the data you
need for your computations:

real colvector y, yhat

y = st_data(., obs, touse)

yhat = st_data(., pred, touse)

With your custom metric function defined in Mata with the signature above, you
can use it as a metric or monitor with validateit by passing the function
name to the metric or monitors options. Note, you will need to make sure
that the function is defined in Mata prior to using it or ensure that it is
defined in a library that Mata will search automatically.

Built-In Metrics and Monitors

The table below lists all of the validation measures that can be used as metrics
or monitors with validateit. The values in the name column below indicate
what to specify in the metric and monitors options to use the
corresponding measures. For additional information about each of the measures
and references, please see libxv.

Name

*** Note this requires installation of polychoric

Binary Metric Details

Multiclass Metric Details

Continuous Metric Details

Examples

Without Monitors

validateit, me(mse) ps(pred) spl(splitvar)

With Monitors

validateit, me(acc) ps(pred) spl(splitvar) mo(npv ppv bacc f1 sens spec)

Returned Values

The following are returned by validateit:

Scalars
r(metric[#]) contains the metric value for the corresponding K-Fold

r(`monitors'[#]) one scalar for each monitor passed to the monitors option, named by the monitor function with a numeric value to identify the corresponding K-Fold

r(metricall) contains the metric value for K-Fold CV fitted to all of the K-Folds

r(`monitors'all) contains the monitor values for K-Fold CV fitted to all of the K-Folds

Matrices
r(xv) contains all of the monitor and metric values

Note, when used with ordinary train/test or train/validate/test splits, no
numeric value will be appended to the name of the returned scalars. The numeric
suffix is only used in the contexts of K-Fold and Leave-One-Out cross-validation.
In the case of K-Fold cross-validation, the numeric suffix indicates the fold
identifier; the number identifies which fold is the held-out set among the
K-folds in the training set. In the case of Leave-One-Out cross-validation, a
value of 1 indicates that the metrics are computed using the held out cases from
the training split.

Additional Information

The display option in validateit is enabled by
collect preview for users running Stata 17 or later.
In addition to providing a convient way for us to structure the display in a
more aesthetically pleasing way, it also makes it easy for you to export the
validation results into any of several formats. The results from
validateit are all stored in the collection named xvval if you do
not pass an argument to the name option. For more information on how to
export these results into the format of your choosing, please see
collect export.

If you have questions, comments, or find bugs, please submit an issue in the
crossvalidate GitHub repository.

Contact

William R. Buchanan, Ph.D.

Sr. Research Scientist, SAG Corporation

SAG Corporation

wbuchanan at sagcorp [dot] com

Steven D. Brownell, Ph.D.

Economist, SAG Corporation

SAG Corporation

sbrownell at sagcorp [dot] com