*! version 0.3.0 07Nov2018

Title

svmachines -- Support Vector Machines

Syntax

svmachines depvar indepvars [if] [in] [, options]

svmachines indepvars [if] [in], type(one_class) [options]

Model
type(svmachines##type:type) Type of model to fit: svc, nu_svc, svr, or nu_svr, or one_class. Default: type(svc)

kernel(svmachines##kernel:kernel) SVM kernel function to use: linear, poly, rbf, sigmoid, or precomputed. Default: kernel(rbf)

Tuning
c(svmachines##c:#) For svc, svr and nu_svr SVMs, the weight on the margin of error. Should be > 0. Default: c(1)

epsilon(svmachines##epsilon:#) For svr SVMs, the margin of error allowed within which observations will be support vectors. Default: eps(0.1)

nu(svmachines##nu:#) For nu_svc, one_class, and nu_svr SVMs, tunes the proportion of expected support vectors. Should be in (0, 1]. Default: nu(0.5)

gamma(svmachines##gamma:#) For poly, rbf and sigmoid kernels, a scaling factor for the linear part of the kernel. Default: gamma(1/[# indepvars])

coef0(svmachines##coef0:#) For poly and sigmoid kernels, a bias ("intercept") term for the linear part of the kernel. Default: coef0(0)

degree(svmachines##degree:#) For poly kernels, the degree of the polynomial to use. Default: cubic (degree(3))

shrinking Whether to use shrinkage heuristics to improve the fit. Default: disabled

Features
probability Whether to precompute for "predict, prob" during estimation. Only applicable to classification problems. Default: disabled

sv(svmachines##sv:newvarname) If given, an indicator variable to generate to mark each row as a support vector or not. Default: disabled

seed(int) Set the seed value. Default: seed(1)

Performance
tolerance(svmachines##tolerance:#) The stopping tolerance used to decide convergence. Default: epsilon(0.001)

verbose Turns on verbose mode. Default: disabled

cache_size(svmachines##cache_size:#) The amount of RAM used to cache kernel values during fitting, in megabytes. Default: 100MB (cache_size(100))

All variables must be numeric, including categorical variables.
If you have categories stored in strings use encode before svmachines.

predict newvar [if] [in] [, options]

probability If specified, estimate class probabilities for each observation. The fit must have been previously made with probability.

scores If specified, output the scores, sometimes called decision values, that measure each observation's distance to its hyperplane. Incompatible with probability.

verbose Turns on verbose mode. Default: disabled

Description

svmachines fits a support vector machine (SVM) model.
SVM is not one, but several, variant models each based upon the principles of
splitting hyperplanes and the culling of unimportant observations.

The basic SVM idea is to find a linear boundary---a hyperplane---in high-dimensional space:
for classification, this is a boundary between two classes;
for regression it is a line near which points should be--much like in OLS,
while simultaneously minimizing the number of observations required to distinguish
this hyperplane.
The unimportant observations are ignored after fitting is done, which makes SVM very memory efficient.

Each observation can be thought of as a vector,
so the support vectors are those observations which the algorithm deems critical to the fit.

This package is a thin wrapper for the widely deployed libsvm.
The thinness of this wrapper is an intentional feature:
it means work done under Stata-SVM should be replicable with other libsvm wrappers such as
Weka or
sklearn.
As a side-effect, some of the options are unfortunately terse.

See the libsvm SVM tutorial for a gentle introduction to the method.
If you find this manual confusing, refer to the authoritative
the libsvm FAQ,
README,
and implementation paper.
Then please write us with your suggestions for clarification.

Please also feel free to send us any other feature requests.

Installation

Since this is just a wrapper, libsvm must be installed to use this package.
On Windows, libsvm.dll is bundled with the package,
and you can find it in your adopath (try findfile libsvm.dll to verify this).
On OS X, libsvm is available in both brew and macports.
On Linux, search for libsvm in your distribution's package manager.
You can also compile and install libsvm from source,
if you cannot find it in your package manager or if you want the latest libsvm.
If you are having plugin load errors, please contact the authors,
as we want to make the experience as smooth as possible for our users across as many platforms as possible.

Options

svmachines

svmachines fits an SVM model, fitting depvar to indepvars except under type(one_class) which only uses indepvars.

libsvm has several algorithms with a single entry point. Since this is a thin wrapper, so do we,
which means not all combinations of options are valid.
Usually libsvm will give an error if you specify an invalid combination,
but sometimes it just ignores parameters without telling you.
Further, amongst valid combinations, not all options and datasets give good results.
Our goal is to have sane defaults,
so that the only choice you usually need to make is what type and kernel to use,
but there is no way to give universal default parameters.

Rather than guessing at the tuning parameters,
you should almost always use cross-validated grid-search to find them.
Which parameters you need to tune depend on which model you pick; for example,
for (type(svc), opt) you only need to find (c(), gamma()).
You can grid-search on a subset of your full data, so long as it is a
representative sample, to quickly find approximations for the optimal parameters.
The libsvm guide explains this in-depth.

type(type) specifies which SVM model to run.

svc and nu_svc perform classification.

depvar should be a variable containing categories.
Multiclass classification is automatically handled if necessary using
the class-against-class method.

If you try to use floating point values with classification you
will find that they are truncated mercilessly to their integer parts,
so you may need to recode your categories before giving them to svmachines.
If you end up with almost as many classes as observations,
you have probably used a continuous depvar and
should use regression instead.

svr and nu_svr perform regression.

depvar should be a variable containing continuous values.

Rather than try to find a hyperplane which separates data as far as possible,
this tries to find a hyperplane to which most data is as near as possible.
See the SVR tutorial for more details.

one_class separates outliers from the bulk of the data.

one_class is a form of unsupervised learning.
It estimates the support of a distribution by distinguishing "class" from "outlier",
based only on the features given to it. Therefore, it does not take a depvar.
Its predictions give 1 for "class" and -1 for "outlier".

Tip: You may use the same varlist as with the other types.
one_class just then interprets your depvar as one of its indepvars,
giving it more information to work with.

To learn about the nu variants, see Chen and Lin's ν-SVM tutorial.

type is case insensitive.

kernel(kernel) gives a kernel function to use.

Much like GLMs, the kernel trick
extends the linear SVM algorithm to be capable of fitting nonlinear data.
Kernels bend a non-linear space into a linear one by applying a high-dimensional mapping.
Under high enough dimensions, any set of data looks close to linear.
See https://www.youtube.com/watch?v=3liCbRZPrZA for a visualization of this process.

The "trick"---and the reason why the set of kernels is hardcoded---is that
for certain kernels the fit can be done efficiently without
actually constructing the high-dimensional points, as the estimation only cares
scoring coefficients u using the output of the kernel, not the output of the values the kernel
is, in theory, operating upon.

Kernels available in this implementation are:

linear: the dot-product you are probably familiar with from OLS: u'*v

poly: (gamma*u'*v + coef0)^degree. This extends the linear kernel with wiggliness.

rbf: stands for Radial Basis Functions, and treats the coefficients
as a mean to smoothly approach in a ball, with the form exp(-gamma*|u-v|^2);
this kernel tends to be a good generalist option for non-linear data.

sigmoid: a kernel which bends the linear kernel to fit in -1 to 1
with tanh(gamma*u'*v + coef0), similar to the logistic non-linearity.

precomputed: assumes that depvar is actually a list of precomputed kernel values.
With effort, you can use this to use custom kernels with your data.

kernel is case insensitive.

c(#) weights (regularizes) the error term used in svc, svr and nu_svr.
Larger allows less error, but too large will lead to underfitting.

epsilon(#) is the margin of error allowed by svr.
Larger makes your fit more able to incorporate more observations, but can lead to underfitting.
Smaller can lead to overfitting.

nu(#) is used in the nu variants.
The nu variants are a reparamaterization of regular SVM which lets you directly tune,
using nu, the size of the svm margin, letting you control over- vs under-fitting.
nu is simultaneously a bound on the fraction of training errors and the
fraction of support vectors. Smaller nu means a smaller margin of error allowed -- so, a tigheter fit -- but more SVs required, and larger nu means a larger margin of error allowed and less SVs required.
See the ν-SVM tutorial for details.

gamma(#) is used in the non-linear poly, rbf and sigmoid
kernels as a scaling factor for the linear part. Larger weights the data more.

coef0(#) similarly is used in the non-linear poly and sigmoid
kernels as a pseudo-intercept term.

degree(#) selects the degree of the polynomial used by the poly kernel.
This literally controls the degree of freedom in the poly fit:
setting this too low results in underfitting and sometimes even non-convergence (notice that at degree(1), this is just the linear kernel);
setting this too high will result in overfitting.

shrinking invokes the shrinkage heuristics,
which can sometimes improve the fit by trading bias for variance.

probability enables the use of "predict, prob".
That does Platt scaling,
so for each class-against-class this precomputes a logistic regression
which is tuned with 5-fold cross-validation.
Internally, libsvm shuffles the data before cross-validation using the OS random number generator,
which is unrelated to Stata's RNG. The seed value for the OS random number generator is set using option seed.
Enabling prob demands a great deal of additional CPU and RAM.

sv(newvarname) records in the given variable a boolean indicating whether each observation was determined to be a support vector.

seed(#) sets the seed value for the OS random number generator. It is not related to the Stata seed.
The seed only affects the results when the prob option is specified. Specifying this seed and the Stata seed enables reproducible results.

tolerance(#) is the stopping tolerance used by the numerical optimizer. You could widen this if you are finding convergence is slow,
but be aware that this usually non-convergence is a deeper problem.
You could also tighten this if you have a powerful enough machine and want to get slightly more accurate estimates.

verbose enables output from the low level libsvm code for the duration of the operation.

cache_size(#) controls a time-memory tradeoff during estimation.
Value is how many megabytes (MB) of RAM to set aside for caching kernel values
Generally, more is faster, at least until you run out of RAM or cause your machine to start swapping.
On modern machines, a reasonable choice is cache_size(1024).

predict

After training you can ask svm to predict what the category (classification) or outcome value (regression)
should be for each given observation. Results are placed into newvar.

newvar must not exist, so if you want to repredict your choices are drop newvar or to pick a new name, e.g. predict newvar2.

For classification (svc, nu_svc) problems, probability requests, for each observation, the probability of it being each class.
newvar is used as a stem for the new columns.
Both probabilities are computed with Platt Scaling. When enabled, so are predictions, and this algorithm is not guaranteed to give the same results
as otherwise. The results should be sensible either way, so if you are getting inconsistent results between the two algorithms,
investigate the tuning parameters.
This option is not valid for other SVM types.

scores outputs the values that svmachines uses to decide on which
side of the hyperplane a particular observation falls. newvar is used as a
stem for the new columns. For type(one_class) and regressions, there is
only one score. For classifications, there is one score for every pair of
classes (this is expensive: k classes means k(k-1)/2 new columns!), because
libsvm aggregates the basic binary-only svmachines algorithm into
a multiclass algorithm with the one-against-one technique. This option is
incompatible with probability because, once trained, the Platt scaling
algorithm does not directly compute scores.

Bug: The score has the wrong sign unless y-values are coded as -1 and 1. For example,
if y is coded as 0 and 1, or 10 and 20, the sign of the score should be reversed.

verbose; see svmachines, verbose.

Prediction implicitly uses the same indepvars as during estimation, so be
careful about renaming or dropping variables.

Stored results

svmachines stores the following in e():

Scalars

e(N) number of observations

e(N_class) number of classes, in a classification problem. 2 in a regression problem.

e(N_SV) number of support vectors.
If e(N_SV)/e(N) is close to 100% your fit is inefficient; perhaps you need to adjust your kernel.

Macros

e(cmd) "svmachines"

e(cmdline) command as typed

e(depvar) name of dependent variable

e(title) title in estimation output

e(model) "svmachines"

e(svm_type) SVM type string, as above

e(svm_kernel) kernel string, as above

e(predict) program used to implement predict

e(levels) list of the classes detected, in the order they were detected. Only defined for type(svc) and type(nu_svc).

Matrices (may be missing)

e(sv_coef) The coefficients of the support vectors for each fitted hyperplane in the dual quadratic programming problem.
Support vectors are only stored if you have less than 11,000 observations or you are using Stata MP.

e(rho) The intercept term for each fitted hyperplane. It is lower-triangular and e(N_class)^2 large, with each entry [i,j] representing the hyperplane between class i and class j.

Remarks

Memory Limits: The cheaper versions of Stata allow only allow fewer variables and smaller matrices to be used.
As machine learning problems typically are on very large datasets,
it is easy to inadvertently instruct this package to construct more columns or larger matrices than you can afford.
If you overflow maxvar, you will receive an error, the operation will fail, and the dataset will be left untouched.
If you overflow matsize, the matrix that overflowed will be missing, but operation will otherwise succeed.

If Stata's memory limits are an impossible hurdle,
your best option is to give up on Stata and switching to libsvm's companion svm-train program.
This will have been installed with the libsvm package if you used a package manager, or
you can get it from its authors;
You can use export_svmlight to extract your dataset for use with svm-train.

Examples

Copyright

The wrapper is licensed:

The MIT License (MIT)

Except where otherwise noted in the code, Copyright (c) 2015
Nick Guenther <nguenthe@uwaterloo.ca>, Matthias Schonlau <schonlau@uwaterloo.ca>

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

libsvm is licensed:

Copyright (c) 2000-2014 Chih-Chung Chang and Chih-Jen Lin
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

3. Neither name of copyright holders nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Authors

Though the license does not obligate you in any way to do so, if you find
this software useful we would be curious and appreciative to hear about your
adventures in machine learning with Stata.

Thank you.

You can contact us at

* Nick Guenther <nguenthe@uwaterloo.ca>

* Matthias Schonlau <schonlau@uwaterloo.ca>

References

Bennett, Kristin P., and Colin Campbell. 2000.
Support Vector Machines: Hype or Hallelujah?
SIGKDD Explor. Newsl. 2.2: 1–13.
http://www.svms.org/tutorials/BennettCampbell2000.pdf.

Chang, Chih-Chung and Lin, Chih-Jen. 2011.
LIBSVM: a library for support vector machines.
ACM Transactions on Intelligent Systems and Technology, 2:27:1--27:27.
http://www.csie.ntu.edu.tw/~cjlin/papers/libsvm.pdf.
Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm

Hsu, Chih-Wei, Chang, Chih-Chung, and Lin, Chih-Jen. April 15, 2010.
A Practical Guide to Support Vector Classification.
http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf.

Smola, Alex J., and Schölkopf, Bernhard. 2004.
A tutorial on support vector regression.
Statistics and Computing 14.3: 199–222.
http://dx.doi.org/10.1023/b:stco.0000035301.49549.88.

Chen, Pai-Hsuen, Lin Chih-Jen, and Schölkopf, Bernhard. 2005.
A Tutorial on ν-Support Vector Machines.
Applied Stochastic Models in Business and Industry 21.2: 111–136.
http://www.csie.ntu.edu.tw/~cjlin/papers/nusvmtutorial.pdf.