*! version 1.1.0 10Feb2020

*! version 1.0.0 07Jan2020

Title

oda -- Optimal Discriminant Analysis

Syntax

oda classvar attributes [if] [in]
, pathcta(filepath)
[ store(filepath)
iter(string)
direction(string)
seed(string)
cat
wt(varname)
missing(integer)
gen(varname)
primary(string)
secondary(string)
name(string)
degen
nopriors
loo
sidak(integer)
]

classvar must be binary; attribute may be binary, ordinal, nomimal, or continuous. If attribute is treated as categorical, it must have more categories than the classvar

pathoda(filepath) specify the file path of the megaODA.exe file; pathoda() is required

store(filepath) specify the file path to store all files and output generated by oda; default is the current working directory

name(string) provide name for all files produced; default is name of the attribute

iter(#) perform # random permutations

direction(#) compute one-sided p-values. "<" or "lt" indicates that the classvar values are ordered in the “less-than” direction. ">" or "gt"
indicates the classvar values are ordered in the “greater-than” direction. The value list must contain every value of the classvar currently defined

seed(#) set random-number seed to #

cat indicates that the attribute should be treated as a categorical variable

wt(varname) weight variable for use in weighted analyses

missing(#) integer value (positive or negative) indicating missing values

gen(varname) a dummy variable indicating two or more independent samples; The gen() solution is a model which, when it is independently
applied to the indicator groups, maximizes the worst performance observed

primary(string) specify the primary criterion for choosing among multiple optimal solutions.
Options are "maxsens", "meansens", "samplerep", "balanced", "distance", "random", or "genmean"; The default is
"maxsens" when priors is on, and "meansens" when the nopriors option is specified

seconday(string) specify the secondary criterion for choosing among multiple optimal solutions.
Options are the same as primary(); The default is "samplerep"

degen allow degenerate cutpoints; default is off

nopriors turn priors off; default is on

loo specify that leave-one-out (jackknife) analysis be performed; loo cannot be specified together with both wt() and cat()

sidak(#) specifies that Sidak adjusted p-values be displayed for p-values derived from the permutation test and loo;
An integer value must be provided to represent the number of comparisons to adjust for

Description

oda is a wrapper program for the Optimal Discriminant Analysis (ODA) software (Yarnold & Soltysik 2005, 2016). Therefore,
ODA must be installed in order for the oda Stata package to work. ODA software is available at https://odajournal.com/resources/.

Optimal discriminant analysis is a machine learning algorithm that was introduced over 25 years ago to offer an alternative analytic
approach to conventional statistical methods commonly used in research (Yarnold & Soltysik 1991). Its appeal lies in its simplicity,
flexibility and accuracy as compared with conventional statistical methods (Yarnold & Soltysik 2005, 2016).

Remarks

To briefly describe how an ODA model is obtained, assume we have a continuous outcome (attribute) and a binary treatment (class) variable.
First, we order the outcome variable from low to high. Next, we find all the points along the continuum of the outcome in which the next
value belongs to an individual from the alternate class than that of the previous value (e.g. the next value belongs to a treated subject,
whereas the previous value belongs to a control). The cutpoint thus represents the mean value of the outcome at this point: cutpoint =
(previous value + current value)/2. Directionality defines how cutpoints are used to classify individual observations. The two directions
are "less than" (controls have lower values on the outcome than treated subjects) and "greater than" (controls have higher values on the
outcome than treated subjects). For an exploratory "two-tailed" hypothesis (controls and treated subjects have different values on the
outcome), both directions are evaluated by the ODA algorithm. For a confirmatory "one-tailed" hypothesis (controls have lower values),
only the appropriate direction (less than) is evaluated. For each cutpoint along the continuum of the outcome, ODA assesses how well
the model – that is, the combination of cutpoint and direction – correctly predicts (in the current example) that controls have values
of the outcome less than or equal to the cutpoint, and treated subjects have values of the outcome greater than the cutpoint (Yarnold &
Soltysik 1991, 2005, 2016).

Optimal discriminant analysis relies on three measures of accuracy to identify the optimal (maximum accuracy) model – that is, the exact
combination of cutpoint and direction that produces the most accurate predictions possible for the sample. Sensitivity or true positive
rate is the proportion of actual treated subjects that are correctly predicted by the ODA model – that is, those who have a value on the
outcome that lies above the cutpoint. Specificity or true negative rate is the proportion of actual control subjects that are correctly
predicted by the ODA model – that is, those who have a value on the outcome that lies at or below the cutpoint. The third measure
of accuracy combines these two metrics and is called the effect strength for sensitivity or ESS (Yarnold & Soltysik 2005, 2016). ESS is a
chance-corrected (0 = the level of accuracy expected by chance) and a maximum-corrected (100 = perfect prediction) index of predictive accuracy. The
formula for computing ESS for a binary (two-category) case classification result is as follows:

ESS = [(Mean Percent Accuracy in Classification – 50)]/ 50 x 100%, where

Mean Percent Accuracy in Classification = (sensitivity + specificity)/2 x 100.

The ODA algorithm iterates through each successive cutpoint and calculates ESS. The maximally accurate model is that which has the cutpoint and direction
with the highest associated value of ESS. Based on simulation research, ESS values <25% conventionally indicate a relatively weak effect, 25-50% indicate
a moderate effect, 50–75% indicate a relatively strong effect and >75% indicate a strong effect (Yarnold & Soltysik 2005, 2016). Optimal discriminant
analysis also computes P-values to assess the statistical reliability (or ‘significance’) of the maximally accurate ODA model. P-values are estimated
using Monte Carlo permutation experiments. For example, in models with a binary treatment, this involves repeatedly shuffling subjects’ treatment
assignment at random, holding their outcome value fixed at its true value. In each permuted dataset, the ESS is recorded, and the permutation P-value
represents the proportion of all permuted datasets in which the ESS is higher than the ESS of the maximally accurate ODA model (Yarnold & Soltysik 1991, 2005, 2016).

Finally, ODA can be implemented using cross-validation to assess the generalizability of the model, using leave-one-out jackknife cross-validation. This typically
entails first estimating a model using a training sample and calculating the accuracy measures, followed by applying the same model to one or more hold-out
(test) samples and then recalculating the accuracy measures. If the accuracy measures remain consistent with those of the original model using the entire sample,
then the model is considered generalizable. This may be important, for example, if the goal of the analysis is to assist researchers identify new candidates for
participation in an ongoing intervention or initiate the intervention in other settings. Cross-validation is less important if the goal is only to estimate treatment
effects of the existing intervention.

Users are strongly encouraged to read Yarnold & Soltysik [2016] for a comprehensive guide to using ODA. Additional references are provided below that detail the use of
ODA for causal inferential problems.

Options

pathoda(filepath) file path where the megaODA.exe is located on the user's computer (e.g. "C:\ODA\"). pathoda() is required.

store(filepath) file path to store all files and output generated by oda (e.g. "C:\ODA\output\"). Default is the current working directory.

name(string) provides a name for all files produced (extensions such as .csv, .out, .txt will be added accordingly). Default is name of the attribute.

iter(#) specifies the number of random permutations to perform. When iter(#) is not specified, permutation P-values are not computed.

direction(string) compute one-sided P-values. "<" or "lt" indicates that the classvar values are ordered in the “less-than” direction. ">" or "gt"
indicates the classvar values are ordered in the “greater-than” direction. The value list must contain every value of the classvar currently
defined, and each character must be separated with a space (e.g. "< 0 1 2" or "lt 0 1 2" or "> 3 2 1" or "gt 3 2 1"). By default, two-sided P-values are
computed (i.e no directional hypothesis).

seed(#) sets the random-number seed in order to generate reproducible permutation P-values.

cat indicates that the attribute should be treated as a categorical variable. The default is to treat the attribute as ordered (continuous).
If cat is specified, the attribute must have more categories than classvar or else an error will be issued.

wt(varname) indicates the weight variable for use with weighted analyses. wt() cannot be the same variable specified as either the classvar
or attribute; ODA treats weight() as an importance weight (see weight).

missing(#) is an integer value (positive or negative) indicating missing values. It is common in surveys to code missing values as 99, 999, 9999, -99, etc.

gen(varname) indicates a dummy variable indicating two or more independent samples; The gen() solution is a model which, when it is independently
applied to the indicator groups, maximizes the worst performance observed amongst all samples.

primary(string) specifies the primary criterion for choosing among multiple optimal solutions.
Options are "maxsens", "meansens", "samplerep", "balanced", "distance", "random", or "genmean". "maxsens" is maximum sensitivity;
"meansens" is the mean of the sensitivities of the separate classes; "samplerep" selects the pattern of predicted class membership most
closely resembling the sample class membership; "balanced" selects the solution in which the sensitivity of the actual classes is most similar amongst each
other; "distance" selects the solution with smallest maximum (over all cutpoints) distance between the cutpoints and their boundaries; "random" selects a
randomly chosen solution; "genmean" is used only when gen is in effect and selects the solution with maximum mean (weighted) sensitivity
over all gen groups. The default is "maxsens" when priors is on, and "meansens" when the nopriors option is specified.

secondary(string) specifies the secondary criterion for choosing among multiple optimal solutions. The choice of criterion is the same
as primary; The default is "samplerep".

degen allows degenerate cutpoints. degen allows flexibility in data sets which have small or no representation in some classes. Conversely, when degen is
not specified, the resulting ODA solution must have at least one observation assigned to each predicted class.

nopriors by default, ODA weights analyses by the reciprocal of sample class membership, in order to adjust for differences in class category sizes. nopriors is
therefore recommended for balanced samples so that no unnecessary weighting will be employed.

loo specifies that leave-one-out (LOO) analysis be performed; In ODA, LOO is not available for weighted categorical problems (i.e. loo cannot be specified together with both wt() and cat()). Additionally,
a P-value for LOO is provided only for a two-category class variable.

sidak(#) specifies that Sidak adjusted p-values be displayed for p-values derived from the permutation test and loo;
An integer value must be provided to represent the number of comparisons to adjust for. iter() must be specified when sidak() is specified. sidak() is currently available with the gen() option.

Examples

(1) Binary class variable and continuous (ordered) attribute

Setup

. sysuse auto, clear

We assess whether ODA can distinguish between foreign and domestic autos based on gas mileage. We perform 1000 permutations and specify that LOO analysis be conducted

. oda foreign mpg, pathoda("C:\ODA\") store("C:\ODA\stuff") iter(1000) loo

(2) Binary class variable and binary attribute

Setup

. sysuse auto, clear

We first generate a binary variable for mpg using the cutpoint of 22.5 derived in Example 1

. gen binmpg = cond(mpg > 22.5,1,0)

We assess whether ODA can distinguish between foreign and domestic autos based on this binary gas mileage variable. We specify that binmpg should be treated as a categorical variable

. oda foreign binmpg, pathoda("C:\ODA\") store("C:\ODA\stuff") iter(1000) loo cat

(3) Binary class variable and multi-category attribute

We assess whether ODA can distinguish between foreign and domestic autos based on levels of repair record ratings. We specify that rep78 is categorical

. oda foreign rep78, pathoda("C:\ODA\") store("C:\ODA\stuff") iter(1000) loo cat

(4) Multi-category class variable and continuous attribute

Setup

. sysuse auto, clear

We first recode missing values to -99

. mvencode rep78, mv(-99)

We assess whether ODA can distinguish between levels of repair record ratings based on gas mileage. We initially run ODA with no permutations or LOO to identify a model

. oda rep78 mpg, pathoda("C:\ODA\") store("C:\ODA\stuff")

Next we specify the directional statement using the values derived from the initial model and add permutations and LOO

. oda rep78 mpg, pathoda("C:\ODA\") store("C:\ODA\stuff") iter(1000) loo dir(< 2 3 1 4 -99 5)

(5) Multi-category class variable and multi-category attribute

Setup

. sysuse auto, clear

We first ensure that missing values are recoded to -99

. mvencode rep78, mv(-99)

Next we create a 3 category variable using the MPG variable

xtile mpg3 = mpg , nq(3)

We assess whether ODA can distinguish between 3 levels of gas mileage based on levels of repair record ratings

. oda mpg3 rep78, pathoda("C:\ODA\") store("C:\ODA\stuff") iter(1000) cat loo

(6) Causal inference with a binary class variable (treatment) and continuous outcome (attribute)

Setup

. webuse cattaneo2, clear

Estimate propensity score for mbsmoke as the treatment, and generate inverse probability of treatment weights (IPTW)

. logit mbsmoke mmarried c.mage c.mage#c.mage fbaby medu

. predict pscore, pr

. gen iptw = cond(mbsmoke, 1/pscore, 1/(1-pscore))

Estimate treatment effect of mbsmoke on bweight using IPTW weights generated in the previous run

. oda mbsmoke bweight, pathoda("C:\ODA\") store("C:\ODA\stuff") wt(iptw) iter(1000) loo

(7) Causal inference with a multi-category class variable (treatment) and continuous outcome (attribute)

Setup

. webuse cattaneo2, clear

Estimate propensity scores for the four treatment levels of msmoke and generate four strata variables corresponding to each treatment, respectively

. mlogit msmoke i.(mmarried fbaby) mage c.mage#c.mage medu c.medu#c.medu c.mage#c.medu, base(0)

. predict double (ps1 ps2 ps3 ps4), pr

Generate weights for nominal treatments on common support using the mmws package (available from SSC).

. mmws msmoke, pscore(ps1 ps2 ps3 ps4) nominal nstrata(5 5 5 5) common replace

Estimate treatment effect of msmoke on bweight using MMWS weights generated in the previous run. Note that we only specify that the model be reported, as this complex analysis is very computer intensive.

. oda msmoke bweight, pathoda("C:\ODA\") store("C:\ODA\stuff") wt( _mmws)

Stored results

oda stores the following in r(), which can be displayed by typing return list after oda is finished. When wt() is specified, the stored values are weighted:

Scalars

r(est_P) Estimated P-value for the permutation test

r(est_adjP) Sidak adjusted P-value for the permutation test

r(est_P_LOO) P-value for the LOO analysis

r(est_adjP_LOO) Sidak adjusted P-value for the LOO analysis

r(ess_train) ESS for the training sample

r(ess_loo) ESS for the leave-one-out analysis (when the loo option is specified)

References

Yarnold PR Soltysik RC. Theoretical distributions of optima for univariate discrimination of random data.
Decision Sciences 1991;22:739–752.

Yarnold PR Soltysik RC. Optimal Data Analysis: A Guidebook with Software for Windows. Washington, DC: APA Books, 2005.

Yarnold PR Soltysik RC. Maximizing Predictive Accuracy. Chicago, IL: ODA Books. DOI: 10.13140/RG.2.1.1368.3286, 2016.

Linden A, Yarnold PR. Using machine learning to assess covariate balance in matching studies.Journal of Evaluation in Clinical Practice 2016;22:848-854.

Linden A, Yarnold PR. Using machine learning to identify structural breaks in single-group interrupted time series designs.Journal of Evaluation in Clinical Practice 2016;22:855-859.

Linden A, Yarnold PR, Nallomothu BK. Using machine learning to model dose-response relationships.Journal of Evaluation in Clinical Practice 2016;22:860-867.

Linden A, Yarnold PR. Combining machine learning and matching techniques to improve causal inference in program evaluation.Journal of Evaluation in Clinical Practice 2016;22:868-874.

Linden A, Yarnold PR. Combining machine learning and propensity score weighting to estimate causal effects in multivalued treatments.Journal of Evaluation in Clinical Practice 2016;22:875-885.

Linden A, Yarnold PR. Using machine learning to evaluate treatment effects in multiple-group interrupted time series analysis.Journal of Evaluation in Clinical Practice 2018;24:740-744.

Citation of oda

oda is not an official Stata command. It is a free contribution
to the research community, like a paper. Please cite it as such:

Linden A. (2020). ODA: Stata module for conducting Optimal Discriminant Analysis. Statistical Software Components S458728, Boston College Department of Economics.

Author

Ariel Linden

President, Linden Consulting Group, LLC

alinden@lindenconsulting.org

Acknowledgments

I wish to thank Paul R. Yarnold for reviewing oda and providing valuable comments.

Also see

Online: cta (if installed), looclass (if installed), kfoldclass (if installed) classtabi (if installed) mmws (if installed)