*! version 1.3 16apr2018

Title

elasticregress -- Elastic net regression

lassoregress -- LASSO regression

ridgeregress -- Ridge regression

Syntax

elasticregress depvar [indepvars] [if] [in] [weight] [, alpha(#) options]

lassoregress depvar [indepvars] [if] [in] [weight] [, options]

ridgeregress depvar [indepvars] [if] [in] [weight] [, options]

Main
alpha weight placed on the LASSO (L1) norm, one minus weight placed on the ridge (L2) norm — by default found by cross-validation;

lambda penalty placed on larger coefficients — by default found by cross-validation;

numfolds number of folds used when cross-validating lambda or alpha — default is 10.

Options which only matter when alpha is found through cross-validation
numalpha number of alpha tested when alpha is found by cross-validation.

Options which only matter when lambda is found through cross-validation
numlambda number of lambda tested when lambda is found by cross-validation;

lambdamin lambda selected is that which minimises cross-validation MSE (default);

lambda1se lambda selected is largest within a standard error of that selected under lambdamin;

epsilon ratio of the smallest lambda tested to the largest — default is 0.001, the user is prompted when this constraint is binding.

Technical options which you can hopefully ignore
tol tolerance on the Euclidean norm of beta;

collinear retain collinear independent variables.

by is allowed; see [D] by.

aweights are allowed; see weight.

Description

elasticregress calculates an elastic net-regularized regression: an estimator
of a linear model in which larger parameters are discouraged.
This estimator nests the LASSO and the ridge regression, which
can be estimated by setting alpha equal to 1 and 0 respectively.

lassoregress estimates the LASSO; it is a convenience command equivalent to
elasticregress with the option alpha(1).

ridgeregress estimates a ridge regression; it is a convenience command equivalent to
elasticregress with the option alpha(0).

Remarks

elasticregress implements the coordinate descent algorithm in Friedman, Hastie
and Tibshirani (2008) to produce very efficient estimates of the LASSO, of the ridge
regression and of a generalization of the two. The algorithm used is the
'covariance updates' algorithm presented in Section 2 of that paper, except that
it does not exploit sparsity in the covariates.

When lambda is not supplied, it is calculated using K-fold cross-validation. In
each cross-validation sample beta(lambda) is calculated using pathwise coordinate
descent — starting at some large lambda, and moving down a sequence of lambda
using the previous solution as the starting vector. Generally the starting lambda
is sufficiently large to guarantee that beta=0, but when alpha is small no such lambda
exists — you will be warned if the upper limit on lambda seems binding. The
sequence of lambda are equidistant in log space and thus the smallest
lambda must be positive. The user will also be warned if this lower limit seems
to be binding (it often is when alpha is small and thus the maximal lambda
is very large).

elasticregress extends the Friedman, Hastie and Tibshirani (2008) by allowing
the elastic-net mixing parameter alpha to be also found using cross-validation.
Consistent cross-validation samples are estimated for a series of alpha, and the
alpha selected is that which minimises CV-MSE(alpha, lambda_min(alpha)), where
CV-MSE() is the mean error in the cross validation samples and lambda_min(alpha)
is the lambda which minimises cross-validation mean squared error given alpha.

To submit bugs or request additional features, please post an issue on the
project Github.

Example

Load our favourite data:

sysuse auto, clear

Set the seed, for consistent cross-validation:

set seed 1

Calculate a LASSO model:

lassoregress mpg weight foreign

LASSO regression Number of observations = 74
R-squared = 0.6466
alpha = 1.0000
lambda = 0.4034
Cross-validation MSE = 12.0339
Number of folds = 10
Number of lambda tested = 100

+

--------

mpg | Coef.

+

--------

weight | -.0054861
foreign | 0
_cons | 37.86231

+

--------

Calculate a ridge-regression model:

ridgeregress mpg weight foreign

Warning: the smallest λ tested was the MSE-minimising λ.
Consider re-running estimation with a smaller epsilon.

Ridge regression Number of observations = 74
R-squared = 0.2343
alpha = 0.0000
lambda = 4.6383
Cross-validation MSE = 27.1411
Number of folds = 10
Number of lambda tested = 100

+

--------

mpg | Coef.

+

--------

weight | -.0010224
foreign | .6956361
_cons | 24.17757

+

--------

Calculate OLS — equivalent to lasso or ridge with lambda=0:

lassoregress mpg weight foreign, lambda(0)

LASSO regression Number of observations = 74
R-squared = 0.6627
alpha = 1.0000
lambda = 0.0000

+

--------

mpg | Coef.

+

--------

weight | -.0065875
foreign | -1.649645
_cons | 41.67843

+

--------

Calculate an elastic-net regression:

elasticregress mpg weight foreign

Warning: the smallest λ tested was the MSE-minimising λ.
Consider re-running estimation with a smaller epsilon.

Elastic-net regression Number of observations = 74
R-squared = 0.6622
alpha = 0.2000
lambda = 0.0232
Cross-validation MSE = 12.8399
Number of folds = 10
Number of alpha tested = 6
Number of lambda tested = 100

+

--------

mpg | Coef.

+

--------

weight | -.0063764
foreign | -1.402116
_cons | 40.96739

+

--------

Stored Results

elasticregress, lassoregress and ridgeregress store the following in e():

Scalars

e(N) number of observations

e(alpha) alpha provided or selected by cross-validation

e(lambda) lambda provided or selected by cross-validation

e(r2) R-squared

e(cvmse_minimal) minimal cross-validation mean squared error

e(cvmse_actual) cross-validation mean squared error for selected lambda (only differs from cvmse_minimal if option lambda1se is used)

e(numfolds) number of folds used in cross-validation

e(numalpha) number of alpha tested in cross-validation

e(numlambda) number of lambda tested in cross-validation

Macros

e(cmd) the command used: elasticregress, lassoregress or ridgeregress

e(varlist_nonzero) list of covariates with non-zero coefficients

e(properties) b

e(depvar) name of dependent variable

Matrices

e(b) coefficient vector

Functions

e(sample) mark estimation sample

Author

Wilbur Townsend

wilbur.townsend@gmail.com

References

Jerome Friedman, Trevor Hastie and Rob Tibshirani. (2008). Regularization Paths for Generalized Linear Models via Coordinate Descent.
Journal of Statistical Software, Vol. 33(1), 1-22 Feb 2010.