*! version 0.0.1 22feb2024

Phases in the Cross-Validation Process

The process of cross-validation follows a standard set of steps to quantify the
out of sample properties of statistical models. The information below describes
this process, the connection to commands included in crossvalidate, and
provides some additional details about existing limitations in the commands
included with crossvalidate.

Splitting the Data

Fitting Models

Predicting Outcomes

Computing the Validation Metric and Monitors

After the Phases are Completed

Once the validation phase is completed the results can be used to support
decisions about using the model to generate predictions (e.g., how well the
model performs on new samples should inform this decision) or can be used for
hyperparameter tuning of models. Hyperparameters are the parameters of the
model that are provided by the analyst/user. For example, setting the tolerance
to determine whether a model using maximum likelihood converges is a
hyperparameter. In other contexts, hyperparameters can include the optimization
method/algorithm used, batch sizes (in the case of batched stochastic gradient
descent), or parameters that determine aspects of the model that affect
complexity/sensitivity (e.g., the number of quadrature points
(see quadchk)).

For users interested in hyperparameter tuning, at this time, you will need to
call the commands splitit, fitit, predictit, and
validateit on your own to implement the updates to the model you are
attempting to tune. Since fitit and predictit both allow you to
pass the estimation command (quoted appropriately), it should not require a
tremendous amount of effort on your part. However, it is possible that you may
want to use a different strategy for the model fitting and predicting phases
(e.g., if you want to adjust hyperparameters after fitting the model to each
k-fold). In those cases, you will need to do a bit more coding.

If anyone using crossvalidate is interested in implementing methods for
hyperparameter tuning, please feel free to reach out to us. We have already
allowed for some option names to be used to define a grid of values, the
parameters to be tuned, and tuning methods in the cvparse function of
libxv.