help for boost(SJ5-3: st0087)

Title

Boosting (Boosted Regressions)

Syntax

boost
varlist
[if] [in] , distribution(string) [options]

Model
distribution(str) possible distributions are normal,logistic(or bernoulli), poisson, and multinomial.

trainfraction(real) percentage of data to be used as training data.By default trainfraction(0.8).

maxiter(int) maximum number of iterations. By default maxiter(20000).

Tuning parameters
shrink(real) specifies the shrinkage factor. By default shrink(0.01).

bag(real) specifies the fraction of training observations that is used to fit an individual tree. By default bag(0.5).

interaction(int) specifies the maximum number of interactions allowed. By default interaction(5).

Other
influence displays the percentage of variation explained.

predict(varname) predicts and saves the predictions in the variable varname.

seed(int) specifies the random number seed to generate the same sequence of random numbers. By default seed(0).

Predict Syntax

predict [if] [in] [, options ]

A name stub can be specified and the * is replaced with numbers 1,2,…, one for each category.

class if distribution(bernoulli), the predictions are rounded to 0 and 1.
If distribution(multinomial) there is no effect;
both class predictions and probabilities are supplied by default.

Post Estimation Command Syntax

influence_delete [, options]

influence_barchart [, options]

Description

boost implements the MART boosting algorithm described in Hastie et al. (2001).
boost accommodates Gaussian (normal), logistic, Poisson and multinomial regression.
The algorithm is implemented as a C++ plugin
and requires Stata 8.1 or higher to run.

By default the model is fit using the first 80% of the data (training data). This
percentage can be changed through the option trainfraction(). To
ensure that the training data are random 80% sort the data in random order
before running boost.

boost determines the number of iterations that maximizes the likelihood,
or, equivalently, the pseudo R squared. The pseudo-R-squared is defined as R2=1-L1/L0 where L1 and L0 are the log likelihood
of the full model and intercept-only model, respectively.
Unlike the R2 given in regress, the pseudo R2 is an out-of-sample statistic.
Out-of-sample R2's tend to be lower than in-sample-R2's.

Options

Model

distribution(str) Possible distributions are
normal,logistic(or bernoulli), poisson, and multinomial. The number of categories in
multinomial is not limited.

trainfraction(real) specifies the percentage of data to be used
as training data. The remainder, the test data
is used to evaluate the best number of iterations.
By default this value is trainfraction(0.8).

maxiter(int) The algorithm stops either when bestiter has not
been reset for 100 iterations or when maxiter is reached. The computing time
is the same whether maxiter=bestiter+100 or maxiter=bestiter+100,000.
When bestiter is too close to maxiter the maximum likelihood iteration may
be larger than maxiter. In that case it is useful to rerun the model
with a larger value for maxiter.
When trainfraction(1.0) all maxiter observations are used for prediction
(bestiter is missing because it is computed on a test data set).

Tuning parameters

shrink(real) specifies the shrinkage factor.
shrink(1) corresponds to no shrinkage.
As a general rule of thumb, reducing the value for shrink
requires an increase in the value of maxiter to achieve a comparable
cross validation R2.
By default shrink(0.01) .

bag(real) Specifies the fraction of training observations that is used to fit an
individual tree. bag(0.5) means that half the observations are
used for building each tree. To use all observations specify bag(1.0).
By default bag(0.5).

interaction(int) specifies the maximum number of interactions allowed.
interaction(1) means that only main effects are fit, interaction(2) means that main
effect and two way interactions are fitted, and so forth.
The number of interactions equals the number of terminal nodes in a tree plus 1.
If interaction(1), then each tree has 2 terminal nodes.
If interaction(2), then each tree has 3 terminal nodes, and so forth.
By default interaction(5).

Other

influence displays the percentage of variation explained
(for non-normal distributions, the percentage of log likelihood explained)
by each input variable for the best number of iterations (only display when the number of variables is less than 20).
If the best number of iterations is 0 (bestiter=0), the influence will be zero for all variables.
For the multinomial distribution, influence is displayed separately for each category.
The influence matrix is saved in e(influence). For the multinomial distribution, column names refer
to category values (with any periods replaced by underscores, because of STATA rules for naming matrix columns.)

predict(varname) predicts and saves the predictions
in the variable varname.
If the distribution is logistic/bernoulli or multinomial, the predicted values are probabilities.
If the distribution is multinomial,
(1) predictions are saved in
multiple variables from varname1 through varname`k', where k is the number of categories
and where variable labels indicate the category that is being predicted, and
(2) the predicted class is saved in varname_class.
To allow for out-of-sample predictions
predict ignores if and in. For model fitting only observations
that satisfy if and in are used, predictions are made for
all observations.
This option also computes {cmd: and e(*accuracy) on training and test data.
Training data refers to the first e(trainn) observations that satisfy [if][in].
Test data refer to the remainder of the obserations that satisfy [if][in].

This option was the original way of specifying predictions.
Prediction can now also be specified using the predict statement following the boost command;
however, then e(*mse) and e(*accuracy) is not computed.

seed(int) seed specifies the random number seed
to generate the same sequence of random numbers. Random numbers are only used
for bagging. Bagging uses random numbers
to select a random subset of the observations for each iteration.
By default (seed(0)). The boost seed option is unrelated to Stata's
set seed command.

Details

The variables may not contain missing values (impute missing values first or drop
observations with missing values. For example, after running a regression the statement
"drop if !e(sample)" would do that.).
When prediction is specified, even the values excluded by [if][in] may not contain
missing values.

The boosting model itself cannot be saved. For this reason predictions is
specified as an option rather than as a post-estimation command. This is different, for
example, for regress where predict can be invoked afterwards.

The number of iterations that boost uses for prediction/influence,
bestiter, cannot be set directly. It is affected indirectly by the choice of
maxiter because bestiter cannot exceed maxiter.

If for logistic regression the train_R2 is missing but the test_R2 is not missing
the test_R2 can be trusted. The missing train_R2 is due to numerical problems in
evaluating the log likelihood functions for very unlikely parameter values. Reset
the number of iterations to bestiter, often this will solve the problem.

The standard output consists of the best number of iterations, bestiter;
the R-squared value computed on the test dataset, test_R2;
the R-squared value computed on the training data set, train_R2
(based on min(maxiter,bestiter+100) iterations)
the number of observations used for the taining data, trainn. trainn
is computed as the number of observations that meet the in/if conditions
times trainfraction(). These statistics can also
be retrieved using ereturn. In addition, ereturn also stores the log-likelihood values from
which train_R2 and test_R2 are computed.

Post Estimation Command Syntax

influence_delete removes variables that were never used in the model, i.e. variables with zero influence as
specified in the influence matrix. This is useful when the number of variables are very large, and the number of
variables in a subsequent run are to be reduced.

min_influence(#) Remove all x-variables that had influence of less than min_influence. By default min_influence = 0, i.e.
only variables with no influence are removed.

influence_barchart
Creates a barchart of the variable influences. There is a separate helpfile for influence_barchart.

Example:Basic prediction example

Put data into random order. Assess contributions of x variables and predict values:

. gen u=uniform()

. sort u

. boost y x1-x7, distribution(logistic) trainfraction(0.8) predict(pred) influence

The boosting implementation currently does not allow missing values. A quick way
to get rid of the missing values to remove all the observations for which the predicted
values after a regression is missing:

. regress y x1-x7

. predict p

. drop if missing(p)

Determine the percentage of correctly classified observations for both the
test and the training data sets:

. global trainn=e(trainn)

. gen class=pred>.5

. gen correct_test= class==y

. replace correct_test=. if missing(y)

. gen correct_train= correct_test

. replace correct_test=. if _n<=$trainn

. replace correct_train=. if _n>$trainn

. tab1 correct_test correct_train y

Display the variable influences in a barchart:

. matrix influence = e(influence)

. svmat influence

. gen id=_n

. replace id=. if influence==.

. graph bar (mean) influence, over(id) ytitle(Percentage Influence)

The bars are labeled with numbers. The corresponding names can be found by typing
matrix list influence.

Example:Prediction with new data

It is currently not possible to save the model, but it is possible to generate predictions for new data.
The model is built excluding data as specified by the "in" or "if" statements,
but the "predict" option ignores "if" and "in". If new data are appended to existing data,
the boost model can be built from existing data, but predictions are
computed for all observations. Here is an example where only the first 1000 observations
are used for model building, but predictions are generated for all observations:

.	boost y x1 x2 x3 x4 in 1/1000, dist(normal) predict

Example:5-fold crossvalidation

in turn use a different 20% of
the data as the test data set and compute an Rsquared value each time. It is assumed
that the boosting plugin is already loaded.

. gen u=uniform()

. sort u

. local N=_N

. local size=round(`N'/5)

. gen group=0

. replace group=1 if _n>`size'

. replace group=2 if _n>`size'*2

. replace group=3 if _n>`size'*3

. replace group=4 if _n>`size'*4

. matrix input R2 = ( )

. forval i=1/5 {

. sort group

. boost y x x2, dist(normal) maxiter(100) trainfraction(0.8)

. replace group= mod(group+1,5)

. matrix R2= R2 \ (e(test_R2))

. }

. svmat R2

. sum R2

Example:Prediction for the Multinomial distribution

The number of prediction variables equals the number of categories.
Each category has a predicted probability.
The predicted category is the category with the largest probability. The code below computes the predicted
category class from the labels of the predicted variables.
(With version 1.3.1 and later such a variable is now created automatically and saved in <varname-class>)

. boost insure age male nonwhite site2 site3 if age!=., dist(multinomial) seed(1) predict(pred) trainfraction(0.8)

. gen class=.

. egen pred=rowmax(pred1-pred3)

. foreach var of varlist pred1-pred3 {

. replace class=`: var label `var'' if pred==`var'

. }

Stored results

boost stores the following in e():

Scalars

e(trainn) Number of observations in the training data (first trainn observations among those that satisfy [if][in])

e(besttiter) Best number of iterations used for fitting

e(test_ll0) log likelihood of the null model in the test data (remainder of observations among those that satisfy [if][in]

e(test_ll1) log likelihood of model with bestiter iterations in the test data

e(train_ll0) log likelihood of the null model in the training data

e(train_ll1) log likelihood of the model with bestiter iterations in the training data

e(train_R2) pseudo R squared in the training data

e(test_R2) pseudo R squared on the test data set (complement of the training data)

e(train_mse) MSE in the training data (normal, poisson only)

e(test_mse) MSE in the test data (normal, poisson only)

e(train_accuracy) Accuracy in the training data (logistic/bernoulli and multinomial only)

e(test_accuracy) Accuracy in the test data (logistic/bernoulli and multinomial only)

Macros

e(predict) program used to implement predict

e(predictlabels) labels used for predicted variables predict

Matrices

e(influence) Influence matrix. Number of rows= number of x-variables. Number of cols=number of y-variables (usually 1)

e(predmat) the matrix of probabilities of labels

References

Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning.
New York: Springer-Verlag; 2001.

G. Ridgeway (1999). The state of boosting. Computing Science and Statistics 31:172-181.

Author

Matthias Schonlau, University of Waterloo
schonlau at uwaterloo dot ca
www.schonlau.net

Also see

Article: Stata Journal, volume 5, number 3: st0087