*! version 1.0 Oct 12 2015 J. N. Luchman
help chaidforest postestimation

Title

chaidforest postestimation -- Postestimation tools for chaidforest

Description

The following postestimation commands are available after chaidforest:

Command Description

predict Obtain predictions across all CHAID trees
estat prox Obtain proximity matrix for estimation sample observations across all CHAID trees
estat fit Cluster/node purity overall fit metric across all CHAID trees
estat import Splitting variables permutation importance metric
estat gettree Obtain the results from a single CHAID tree from the forest

The above postestimation commands require the chaidtree vector object "results" to be in memory in Mata. chaidforest will have to
be re-run if any command cannot find the results chaidtree vector object as all tree results are stored as instances of that class.

Syntax for predict

predict newvar [, predict_options]

predict_options Description

Forest-level
pr Empirical probabilities or observed proportions across all CHAID trees
mode Most likely or modal value across all CHAID trees
CHAID tree-level
ctpr Empirical probabilities or observed proportions for any given CHAID tree
ctmode Most likely or modal value across for any given CHAID tree
option
useboot Use bootstrap weights when computing empirical probabilities or modal value for each CHAID tree

pr and ctpr are defaults, are always generated as doubles, and the newvar is interpreted as a stub.
mode is always generated as a long.
All predictions are generated for the entire estimation sample.

Options for predict

pr, the default forest-level prediction option, computes empirical probabilities for each category of the response variable.
How the probabilities are computed depends on the data generated at the CHAID tree-level (discussed below in remarks). Missing values
are a valid category if missing was invoked in chaidforest producing a predictied probability of ".p"
(i.e., predicted missing). pr cannot be used with mode.

mode is the category of the response variable which is deemed most likely or receives the most "votes" for an observation
across all the CHAID trees. As with the pr option, how the modal value is computed depends on how the data are generated at the
CHAID tree-level (ties for most likely value are decided randomly). Missing values are a valid prediction if missing was
invoked in chaidforest producing a predicted ".p" (i.e., predicted missing). mode cannot be used with pr.

ctpr, the default CHAID tree-level prediction option, computes empirical probabilities for each category in the response variable for
a specific CHAID tree. Specifically, the clusters/nodes produced by each tree receive their own tabulation. The proportion in each of the
categories in the response variable are then assigned as predicted probabilities of being in each category to all the observations in that cluster.
The empirical probabilities produced in the CHAID tree-level step are then aggregated across the forest by the forest-level commands for each
observation. Missing values will be incorporated at the CHAID tree-level if missing is specified. ctpr cannot be used
with ctmode.

ctmode is the category of the response variable which is deemed most likely or has the highest frequency for a cluster within a CHAID
tree. The process to obtain a most likely value mirrors that to obtain empirical probabilities except that only the most likely value
is retained (ties for most likely value are decided randomly). The most likely values produced in the CHAID tree-level step are
then aggregated across the forest by the forest-level commands for each observation. Missing values will be incorporated at the
CHAID tree-level if missing is specified. ctmode cannot be used with ctpr.

useboot uses the bootstrap weights invoked to generate the CHAID trees to obtain empirical probabilities and most likely values
at the CHAID tree-level. Currently, the useboot option is experimental and will generally result in sub-optimal fit to the observed
data.

Remarks on predict

chaidforest always uses one option from both the "forest-level" and "CHAID tree-level" categories. Thus 4 different types of predicted
values are possible to obtain with chaidforest.

Arguably, the most nuanced and complete predictions are obtained from the default pr-ctpr combination where empirical
probabilities for each category of the response variable are obtained for each observation in each CHAID tree and averaged across
the entire forest to result in a single set of empirical probabilities for each category of the response variable for each observation.
chaidforest version 1.0's default was more akin to a pr-ctmode approach where a single, modal predicted
value was obtained for each observation in each CHAID tree. Empirical probabilities for each obervation then represent the
proportion of each category of the response variable observed for each observation as aggregated across all CHAID trees in the forest.

The forest-level mode options produce results more akin to those from other random forest implementations as only a single value is produced.
The somewhat more nuanced mode-ctpr approach first obtains empirical probabilities of each category of the response variable for
each observation from each CHAID tree and averages them across the entire forest. The most likely value of the response variable for each
observation is then determined and used as the predicted value. By contrast, the mode-ctmode is the most simple approch to obtaining
a predicted value, is the approach most commonly used by other random forest algorithms, and is the approach used by chaidforest version
1.0 when using mode. Here each CHAID tree supplies a predicted value for each observation. The final predicted value is the
category with the most votes or the highest frequency across all CHAID trees in the forest.

Syntax for estat prox

estat prox [, mata]

Options for estat prox

mata overrides default behavior saving the proximity (or more accurately dissimilarity) matrix to the ereturned matrix
list in favor of saving the proximity matrix computed by estat prox in Mata as a matrix named prox. Note that any Mata
matrix named prox will be overwritten. The mata option is useful for large datasets.

Saved results for estat prox

estat prox saves the following results to e():

matrices

e(prox) N x N matrix of dissimilarities between observations

Remarks for estat prox

The proximity matrix returned by estat prox represents a metric for assessing similarity between the observations as impled by the
chaidforest results. As such, the proximity matrix produces a matrix which can be used by mdsmat and clustermat. The
proximity matrix thus allows for clustering observations based on aggregated chaidforest model results which can facilitate interpretation
of the results.

estat prox proceeds by obtaining the proportion of CHAID trees in which each pair of observations falls into the same cluster.
In the instance, that 2 observations are always in the same cluster in every tree in the forest, they would obtain a value of 1. Conversely,
2 observations which are never in the same cluster in every tree in the forest would obtain a value of 0.

The proximity matrix is actually a dissimilarity matrix as the complement of each of the proportions (i.e., 1 - proportion) are used
as the results in the proximity matrix. All entries in the matrix are als square rooted to improve the performance of the matrix in other programs.

Syntax for estat fit

estat fit [, insample]

Options for estat fit

insample overrides the default resampling-based fit metric and instead uses the bootstrap weights that were used to estimate
the chaidforest results. Thus, the overall fit is based in fully in-sample weights for each CHAID tree. If proos() was used in the
base chaidforest command, only the observations sampled are used to compute the fit metric. If nosamp was used in the base
chaidforest command, all observations in the estimation sample are used.

Saved results for estat fit

estat fit saves the following results to e():

scalars

e(fit) Cramér's V-based fit metric

Remarks for estat fit

The fit metric returned is based on the average Cramér's V from each CHAID tree across the entire forest. By default,
estat fit obtains a different bootstrap sample of observations than the original bootstrap sample that chaidforest
used to estimate the CHAID tree, applies the cluster/node rules to the resampled data, and computes Cramér's V using the
new bootstrap sample. Hence, the fit metric is not an out-of-sample fit metric, but mimics one as its a resample fit metric
which will use some of the same observations as in the original bootstrap sample, but also some from out-of-sample, and has
an entirely new set of bootstrap weights. In both cases, the total sample used to compute fit will look different than the
original and serves as a useful assessment of how well the results will hold up in replication.

When using the proos() option in the base chaidforest command, the resample fit metric will mimic the original process,
resampling the proportion desired without replacement. The nosamp option in the base chaidforest command does not
affect estat fit without the insample option. The insample option will obtain the fit metric across all CHAID
trees in the forest without resampling. In the absence of the insample option, the fit metric will resample observations in each
CHAID tree.

Syntax for estat import

estat import [, insample]

Options for estat import

insample identical to the behavior in estat fit above.

Saved results for estat import

estat import saves the following results to e():

matrices

e(import) matrix of permutation importance values based on Cramér's V-based fit metric

Remarks for estat import

The permutation importance values are computed using the method outlined in estat fit. The importance values differ from
the traditional fit values in that for every splitting variable the data for that variable is randomly permuted or jumbled
for each CHAID tree - which is done before any resampling as is described in estat fit.

After permuting, the CHAID tree splitting rules are applied to the permuted data, and the fit metrics (the entries in the second
row of e(import)) are conputed. The fit metrics are then rank ordered (the first row of e(import)). Splitting variables
with smaller fit metrics are deemed more important as fit is reduced more when their values are permuted - thus, by implication, they
contribute more to the fit metric.

Syntax for estat gettree

estat gettree, tree(tree_number) [graph]

Options for estat gettree

tree(tree_number) a required option specifying the results from which tree should be pulled from the chaidtree object associated with
the most recent chaidforest run.

graph mimics chaid's behavior in graphing and displaying the tree structure if the focal CHAID tree from the
chaidtree object associated with the most recent chaidforest run.

Saved results for estat gettree

estat gettree saves the following results to r():

scalars

r(N_clusters) number of clusters created by chaid and returned in _CHAID_tree_number variable

r(fit) purity of the clusters (extent to which each cluster has only a single value of the response variable);
based on Cramér's V

macros

r(splitvars) displays splitting variables selected for use in the current CHAID tree

r(path#) displays the levels of each split leading to cluster #. Each split is separated by semicolons.
The splitting variable is first followed by an "at" sign (i.e., @) followed by the levels of the splitting variable that
describes cluster # at that split

matrices

r(sizes) sample size of each cluster

r(branches) number of branches from the root node for each cluster

Remarks for estat gettree

Generates 2 variables associated with the CHAID tree tree_number in the forest. The first _CHAID_tree_number corresponds
to cluster membership for each observation in the dataset for CHAID tree tree_number.

The second variable bwgt_tree_number is the bootstrap frequency weight obtained for CHAID tree tree_number.

estat gettree is intended to "open up the black box" of the random forest algorithm to see individual trees' results.