*! version 0.0.1 10May2017

Title

par

gridsearch -- Optimizing tuning parameter levels with a grid search

Syntax

gridsearch command depvar indepvars [if] [in] , method(str1 str2) par1name(str) par1list(numlist) criterion(str) [ options ]

gridsearch discrim subcommand indepvars [if] [in] , method(str1 str2) par1name(str) par1list(numlist) criterion(str) group(depvar) [ options ]

par1name(string) Name of the a tuning parameter of command

par1list(numlist) Values to explore for tuning parameter

par2name(string) Name of the an optional second tuning parameter of command

par2list(numlist) Values to explore for the second tuning parameter

criterion(string) Evaluation criterion

method(str1 str2) str1 specifies train-validation method; str2 specifies corresponding option.

nogrid Explore all parameter values as a list (do not form a grid)

options Additional options are passed to the estimation command

predoptions(string) Any prediction options are passed to the prediction command

Description

gridsearch runs a user-specified statistical learning (aka machine learning) algorithm repeatedly with a grid of values corresponding to one or two tuning parameters. This facilities the tuning of statistical learning algorithms.
Examples of statistical learning algorithms that require tuning include support vector machines, gradient boosting, k-nearest neighbors, random forests.

After evaluating all combinations of values according to criterion, gridsearch lists the
best combination and the corresponding value of the criterion. Running time (in seconds) is also given.

gridsearch may be very time intensive depending on the estimation command and the number of parameter
combinations to explore.

Remarks

Only estimation commands that allow the use of predict after the estimation command can be used.
The program does not currently support the prediction of multiple variables as would be needed, for example,
for multinomial logistic regression.

The discrim command has a slightly different syntax and is listed separately above. It is particularly useful for
k nearest neighbor : discrim knn.

Options

par1name(string) Name of a tuning parameter of command.

par1list(numlist) Values of the tuning parameter to explore.

par2name(string) Name of an optional second tuning parameter of command.

par2list(numlist) Values of the second tuning parameter to explore.

criterion:(string) Implemented criteria for binary outcomes (classification) is "accuracy" and "AUC" (area under the curve)
and for continuous outcomes (regression) is "mse". If trainfraction is specified,
evaluation occurs on the test observations. If cv is specified evaluation occurs on all observations.
For classification, if predict gives a probability rather than 0/1 classification
the probabilities are rounded.

method(str1 str2) str1 specifies the the method, str2 the corresponding option.
Available methods are "trainfraction", "trainvar" and "cv".

method(trainfraction #) Fraction of the data used for training. The first (trainfraction*N) observations
are designated training data, the remainder the test data. To ensure that training data are selected at random,
the data should be in random sort order before calling gridsearch. trainfraction should be in the range of (0,1).

method(trainvar varname) Indicator variable (0,1) for whether or not the data are training data.
This is useful when training observations cannot be sampled at random.
For example, if there are multiple measurements for each person, all measurements for one person should be either in the training or in the test data.

method(cv #) Number of folds for crossvalidation. This option requires our command crossvalidate which must be installed first. This option can be very time consuming.

nogrid Explore parameter values as a list: the i th value of the first tuning parmaeter is paired with the i th value of the second and third tuning
parameter. If not specified, explore all combinations of the values of the tuning parameters (grid).
This option has no affect if only one tuning parameter is specified.

options Additional options are passed to the estimation command. The same option
must not be specified twice: tuning parameters must not also be specified as an additional option.

predoptions(string) Any prediction options are passed to the prediction command. This is rarely used but useful, for example, for randomforest
where predict requires specification of either class or reg.

Examples

Example k nearest neighbor classification

k nearest neighbor classification. k is the only tuning parameter. Search for values of k=1,2,...,5.

. sysuse auto

Shuffle

. set seed 793742

. generate u = runiform()

. sort u

. gridsearch discrim knn weight length mpg price , par1name(k) par1list(1/5) group(foreign) criterion(accuracy) method(trainfraction .8)

Gridsearch outputs the best parameter combination. Optionally, can look at the accuracy of all runs:

. matrix gridsearch= r(gridsearch)

. svmat gridsearch , names(col)

. list accuracy k seconds in 1/5

Example support vector machines regression

We consider two tuning parameters, c and gamma. We first standardize the variables. This is a toy
example: the sample size, 74 observations, is too small for a training/test split.
svmachines is a user written programand and must be downloaded first via net search svmachines.

. sysuse auto

. foreach var of varlist weight length mpg headroom {

. sum `var'

. replace `var'= (`var'-r(mean))/r(sd)

. }

. gridsearch svmachines price foreign weight length mpg headroom , par1name(c) par1list(0.1 1 10 100 1000 10000) par2name(gamma) par2list(1 5 10) criterion(mse) method(trainfraction .5) type(svr) kernel(rbf)

Example (continued): split into train/validation/test

The previous example used trainfraction to split the data into training and validation data and evaluated the gridsearch on the validation data. Instead, we now wish to split the data into train/validation/test data. We validate the gridsearch on the validation data, without ever using the test data.

. set seed 793742

. generate u = runiform()

. sort u

. gen test= _n> 0.8*_N

. gridsearch svmachines price foreign weight length mpg headroom if !test, par1name(c) par1list(0.1 1 10 100 1000 10000) par2name(gamma) par2list(1 5 10) criterion(mse) method(trainfraction .5) type(svr) kernel(rbf)

Example (continued): Use crossvalidation

Instead of splitting the data into training/validation data, we can also use cross-validation.

. gridsearch mypred svmachines price foreign weight length mpg headroom, cv(5) par1name(c) par1list(0.1 1 10 100 1000 10000) par2name(gamma) par2list(1 5 10) criterion(rmse) type(svr) kernel(rbf)

As before if !test can also be specified (not shown here).

Stored results

Matrices

r(gridsearch)
Matrix with one row for each search
and column names `par1name', `par2name' (if specified), `criterion', seconds

Scalars

r(tune1)
Best value according to the criterion chosen for tuning parameter 1

r(tune2)
Best value according to the criterion chosen for tuning parameter 2

Author

Matthias Schonlau <schonlau@uwaterloo.ca>