*! miinc version 2.0 August 7, 2014 J. N. Luchman
help mmiinc

Title

Multi-model inference using information criteria (MIINC)

Syntax

miinc [se] depvar [indepvars] [if] [in] [weight] [, reg(estcmd, regopts)
ic(aicc | aic | bic | noic) bestmodels(#) pip sets((varlist) (varlist) ...)
all(varlist) ttest ll(scalar) parm(scalar) obs(scalar)]

or

miinc me [(eqname1 = indepvars) (eqname2 = indepvars) ...] ,
reg(full estcmd with comma) [ic(aicc | aic | bic | noic) bestmodels(#) pip
sets((parmlist) (parmlist) ...) ttest ll(scalar) parm(scalar) obs(scalar)]

mi estimate is allowed (see mi estimate); see prefix. pweights, aweights, iweights, and
fweights are allowed but must be able to be used by the estimation command in reg(), see help weights, and must
be included in the reg() option with miinc me. Factor variables are also allowed, but like weights, must be accepted by
the estimation command in reg(). svy is allowed (see svy); but only as a prefix to the command within reg().

Description: Basic Information and Use

miinc is a suite of utility commands for model selection, model averaging, and independent variable/parameter inclusion probability
using information criteria as the basis for model fit. The framework implemented in miinc derives from Burnham and Anderson's
(2002; 2004) work on multi-model inference. A model, in the present case, refers to a set of independent variables or parameter estimates.

miinc has 2 different modes. The first mode, miinc se, is the single equation version of miinc. miinc se is for use with
estimation commands that follow the traditional depvar indepvars format common to most single equation commands. miinc se
is the default version of miinc and does not require the se suffix. The second mode, miinc me, is the multiple equation
version of miinc. miinc me is more flexible than miinc se as it allows for multiple equations of any form, but does require that
the estimation command in reg() accept constraints. All commands to be used by miinc must produce and store some sort of
log-likelihood, parameter estimate count, and number of observations. miinc's defaults are to look for e(ll) as a log-likelihood,
e(rank) as a parameter estimate count, and e(N) as a number of observations count to derive information criteria.

The miinc modes operate differently in terms of how they implement multi-model inference, which requires some description. miinc se
operates by treating all the variables in the indepvars list as independent variables about which the user is uncertain and would like to
model average. Additionally, all the independent variables bound together in the same or separate sets by the sets() option are considered
to be uncertain, but should be included together in any particular model. Thus, independent variables bound together in a distinct sets()
set have joint (un-)certainty. Finally, independent variables in the all() option are those about which the user has certainty and are
thus included in every/all model(s). By default constants are included in every model. miinc se uses the estcmd or estimation
command specified in the reg() option by substituting everything on the left-hand side of the comma in the reg() option
(including prefixes) at the beginning of the command. miinc se puts everything to the right-hand side of the comma in reg() at the
end of the command as options or regopts. The comma in the reg() option for miinc se is optional (as the user may not have
regopts to pass to the estcmd. In fact, reg() is itself optional as the default estcmd is regress with all of
regress' command defaults.

miinc me forces the user to specify a full estcmd in reg(). That is, the full estcmd requires that the user write out
a full command (prefixes [other than mi estimate], command name, model components, and options) to be interpreted by Stata and would be
interpreted as an estimation command if inputted to Stata outside of the reg() option. miinc me also requires a comma for options to
be included in reg() whether or not any options are included as the multi-model inference is conducted using constraint. Given the
full estcmd in reg(), miinc me parses all the equations enclosed in parentheses before the comma, building a parmname by
linking the eqname on the left-hand side of the equal sign with each separate independent variable in the associated indepvars varlist.
Thus, miinc me creates a set of parmnames of the form "eqname:indepvar" about which the user is uncertain in the full estcmd
in reg(). The sets() option operates similarly, except that the user must write out each parmname in the form
"eqname:indepvar". Thus, a parmlist can be generated by enclosing sevaral parmnames in parentheses to bind. Just as in
miinc se, the parmnames bound together by parentheses are considered jointly (un-)certain. miinc me has no all() option as
all parameters not either specified in the equations before the comma or in the sets() option are considered to be certain and are included
in every model.

Description: Technical

In terms of model selection, miinc's approach stems from of information theory. In particular, miinc uses the Kullback-Liebler
divergence estimate from information theory as represented by the Akaike (aic and small-sample corrected aicc) or Bayesian (bic) information
criterion. Models are ranked based on the focal information criterion and can be returned using the bestmodels() option. As is
noted by Burnham and Anderson (2004; p. 271), differences between 0 and 2 from the smallest information criterion
producing/best model can be interpreted as having strong support, differerences between 4 and 7 can be interpreted as
having coniderably less support than the best model, and differences greater than 10 have essentially no support compared to
the best model.

miinc's main focus is on model averaging. The model coefficients are averaged based on weights derived from the focal
information criterion following Burnham and Anderson (2002; 2004). Specifically, each model's information criterion is subtracted from the
smallest information criterion obtained across all models i (obtaining IC_difference_i). Each model's information criterion is then
transformed back into a likelihood of the model given the data (i.e., adj_lhd_i = exp(-2*IC_difference_i)). Information criterion
weights are then obtained for each model by forming a ratio of adj_lhd_i to the sum of all the adj_lhds across all models to obtain
IC_weight_i. Finally, the model average coefficient vector is the sum of the product of IC_weight_i and the coefficient
vector for model i across all models (see Burnham & Anderson, 2004; Equation 1). The model average variance-covariance matrix for the
estimator is similarly derived. Specifically, the model averaged sampling variance-covariance matrix is obtained by summing the product of of
IC_weight_i and the sum of model i's sampling (co-)variance estimate and the squared (for a sampling variance) or the product of the
differences (for a sampling covariance) between the model averaged coefficient(s) and model i's estimate of that/those coefficient(s),
summed across all models (see Burnham & Anderson, 2004; Equations 4 & 5).

miinc can also produce estimates of independent variable/parameter inclusion probability. The approach taken by miinc mirrors
the posterior inclusion probability metric produced by many Bayesian model averaging routines (see findit bma). Thus, the inclusion
probability values produced by miinc are the sum, across all models, of the IC_weights (see Burnham & Anderson, 2004; p. 274).

miinc prodeeds by first estimating coefficients and samping variance-covariances from the specified model for each valid subset of the
independent variables or parameters about which the user has uncertainty. miinc is factor variables "smart." That is, miinc
does not allow inapproproate models to be estimated and averaged. For models with dummy codes/indicators (i.e., factor variables with
"i."), all indicators associated with an independent variable for miinc se are bound together and always included in the same models
like a set from sets(). By contrast, with miinc me parameters with the same equation and independent variable need to be bound
together explicitly in a sets(). Hence, all the dummy codes/indicators are and need to be considered as a set. For models with exponents
or interactions (i.e., factor variables with "#" or factorials "##"), all lower-order or marginal terms must be present when a
higher-order term is present in a model. miinc drops all models that include higher-order terms but do not include lower-order or marginal
terms prior to all subsets estimation. Note: factor variable specifications of the type i(# #..#).varname are not allowed in
miinc se's varlist as the spaces in the factor variable notation disrupt syntax parsing. Specific levels should be included explicitly
in sets() as #.varname #.varname..#.varname.

Options

reg(estcmd, regopts) (for miinc se) specifies the regression command to be model averaged - which can include
any user-written program. As applied to miinc se user-written programs must follow the traditional cmd depvar indepvars syntax.
reg() also allows the user to pass options for the estimation command used by miinc. When a comma is added in reg(), all the
syntax following the comma will be passed to each run of the estimation command as options. miinc se defaults to reg(regress) and will
produce a warning.

reg(full estcmd with comma) (for miinc me) specifies the full command to be interpreted by Stata and model averaged -
which can also include any user-written program. reg() is required for miinc me and requires a comma to be included in the command
if there are not any regopts specified in the full estcmd in option reg(). A comma must be included as miinc me uses the
contents of reg() are used as: "contents of option reg()" constraints(numlist) directly in Stata's command prompt.
Thus, without a comma, the constraints(numlist) is interpreted as a part of the equation specification (which will produce an error).
Note that other constraints cannot be used with miinc me currently.

ic(aicc | aic | bic | noic) specifies the information criterion that miinc uses for model selection, averaging, and inclusion
probability. miinc currently has 4 options for information criteria. The aic computed as -2*(ll)+2*(parm).
The bic computed as -2*(ll)+(parm)*ln((obs)). The aicc (the small sample corrected aic) computed as
-2*(ll)+2*(parm)+(2*(parm)*((parm)+1)/((obs)-(parm)-1)). Finally, ic(noic) is a non-informative or equal weighting approach and
only produces model averaging (i.e., cannot rank models/independent variables and thus will not produce a set of best models with
option bestomdels() or inclusion probabilities with pip). miinc defaults to ic(aicc) and will produce a warning.

sets() binds together independent variables or parameter estimates as a set in the each subset estimation. Hence, all variables in
a set will always appear together in a model.

The user can specify as many sets of arbitrary size as is desired and the basic syntax follows: (for miinc se)
sets((x1 x2) (x3 x4)), and (for miinc me) sets((eq1:x1 eq1:x2) (eq2:x3 eq3:x4)); this will create
two sets. The first set (denoted "set1" in the results) will created from the variables x1 and x2 (or parameters eq1:x1 and eq1:x2) and the
second set (denoted "set2" in the results) will be created from the variables x3 and x4 (or parameters eq2:x3 and eq3:x4). All sets must be
bound by parentheses - thus, each set must begin with a left paren "(" and end with a right paren ")" and all parentheses separating sets in
the sets() option syntax must be separated by at least one space. Independent variables in a set will receive identical information
criterion weights (when using option pip) and will always be selected together (if using option bestmodels()).

all() (for miinc se) includes a set of independent variables in all models. Thus, all independent variables included in the all() option
appear in all the models estimated by miinc. Thus, the all() option allows the user to incorporate variables about which they are
more certain in terms of their inclusion in the model. Consequently, variables in the all() option do not produce informative
information criterion weights (i.e., if the pip option is used as they will always be 1).

all() (for miinc me) is not a valid option as all parameters not included in the equations before the comma
or the sets() option in miinc me are considered to be all() parameters and will not produce informative information
criterion weights (i.e., if the pip option is used as they will always be 1).

bestmodels() tells miinc to return the number of models with the lowest information criterion differences. The matrix returned
by bestmodels() has the information criterion differences in its first column. Each column that follows is associated with a coefficient
from the e(b) or coefficient vector matrix. Missing values in the bestmodels() matrix indicate that the parameter associated with the
column in question was not in the model associeted with the row in question.

pip tells miinc to return a vector of inclusion probabilities for the independent variables. The pip vector is a
"weight of evidence" metric and shows reflects how much confidence the user can have in the the inclusion of the independent variable in the model.
The pip metrics are, of course, relative to the best model under consideration and, consequently, do not guarantee that highly likely variables
are necesarily in the true model. Rather, the pip metric} simply suggests the usefulness of the independent variable (which is sometimes
taken as an indication of variable importance). Interaction and non-linear terms. owing to their conditioning on having the lower-order or
marginal terms in the model, will produce pip values that are bounded by their lower-order terms.

ttest changes the test statistic distribution each coefficient in e(b) is tested against from the normal (i.e., "z") to the
Student's t (i.e., "t") distribution, as is displayed in the output. Useful for estimation commands that test against the Student's t such as
regress.

ll() changes the scalar (pseudo-)log-likelihood value miinc looks for and substitutes into (ll) in the information
criterion computations (see entry for option ic() above). The default scalar miinc searches for is e(ll) (see ereturn).

parm() changes the scalar parameter estimate count miinc looks for and substitutes into (parm) in the information
criterion computations (see entry for option ic() above). The default scalar miinc searches for is e(rank) (see ereturn).

obs() changes the scalar observations count miinc looks for and substitutes into (obs) in the information
criterion computations (see entry for option ic() above). The default scalar miinc searches for is e(N) (see ereturn).

Final Remarks

miinc se and miinc me are a very flexible convenience commands that can incorporate existing commands (by Stata Corp or
user-written) to model average so long as they follow the traditional depvar indepvars format (for miinc se) or allow
constraints (for miinc me). Thus, most any program estimated by ml or nl could be model averaged using miinc.

For user-written commands or wrappers, miinc se requires that the program accept at least one optional argument and must accept an
if statement in its syntax line. miinc me has no such requirements, but must be compatible with constraint (thus requires
ml as estimator).

miinc stores the ereturned results from the full model in memory and adds to/replaces those results in memory. Thus, most
postestimation commands that work with the base estimation command will work with the model-averaged results returned by miinc (e.g.,
margins, test, predict, etc.)

miinc se and miinc me do not directly accept the svy prefix - but do accept the svy prefix within the reg()
option. When miinc se and miinc me detect the svy prefix, the information criterion automatically changes to ic(aicw)
(unless ic(noic) is specified) or the rao-scott adjusted AIC and will check the command in reg() to determine whether the command returns
the necessary matrices (i.e., e(V), e(V_srs), [both needed for the rao-scott adjustment] and e(ilog) [to obtain the
pseudo-log-likelihood]; see svy: tabulate twoway for an discussion of the adjustment; see also Scott, 2013; for the
conceptual foundation and example #7). In addition, miinc se can model average xtgee models using the user-written
qic command/metric. To use the qic metric, the qic command must be installed
and it must be included in the (opt reg() option (see example #9). When miinc se detects qic, it automatically switches to
ic(qic) (again, unless ic(noic) is specified). The qic command will not work properly with miinc me and should only be
used with miinc se.

Introductory examples

webuse auto

Example 1: default linear regression-based model averaging with and without the (optional) "se" suffix

miinc price mpg rep78 headroom, ttest

miinc se price mpg rep78 headroom, ttest

or

miinc me (price = mpg rep78 headroom), reg(sureg (price = mpg rep78 headroom), dfk2) ttest

Note: cnsreg and glm do not work with miinc me due to incompatibility in the way miinc me
creates constraints and how both commands use contsraints.

Example 2: Ordered outcome model averaging with certain "turn" variable and the BIC

miinc se rep78 trunk weight length, reg(ologit) ic(bic) all(turn)

or

miinc me (rep78 = trunk weight length), reg(ologit rep78 trunk weight length turn,) ic(bic)

Example 3: Poisson model averaging with factor varaible and set

miinc se price weight i.rep78, reg(poisson) ic(aicc) sets((trunk turn))

or

miinc me (price = weight), reg(poisson price weight ib1.rep78 trunk turn,) ic(aicc)
sets((price:trunk price:turn) (price:1b.rep78 price:2.rep78 price:3.rep78 price:4.rep78 price:5.rep78))

Example 4: Model averaging with interaction and non-linear variables showing 10 best models

miinc se price c.mpg##c.headroom c.mpg#c.mpg c.headroom#c.headroom, reg(regress) bestmodels(10) ttest ic(aicc)

Example 5: Model averaging with posterior inclusion probability and bootstrap standard errors

miinc se price mpg headroom trunk turn gear_ratio foreign length weight, reg(regress, vce(bootstrap)) ttest ic(aicc) pip

Example 6: Seemingly unrelated regression model averaging

miinc me (price = mpg turn) (gear_ratio = displacement), reg(sureg (price = mpg turn) (gear_ratio = displacement),) ic(aicc)

Example 7: Survey regression with rao-scott adjusted AIC

webuse nhanes2f, clear

svyset psuid [pweight=finalwgt], strata(stratid)

miinc se zinc age age2 weight female black orace rural, reg(svy: glm)

Example 8: Multiply imputed model averaging

webuse mheart1s20, clear

mi estimate: miinc se attack smokes age bmi hsgrad female, reg(logit) ic(aicc)

or

mi estimate: miinc me (attack = smokes age bmi hsgrad female), reg(logit attack smokes age bmi hsgrad female,) ic(aicc)

Example 9: Generalized estimating equation model averaging

webuse nlswork, clear

xtset id year

miinc se union c_city not_smsa collgrad tenure, reg(qic)

Saved results

miinc adds/replaces the following results in e():

scalars

e(bestic) information criterion value associated with the best model

macros

e(cmdline) command as typed

e(title) Multi-model inference

e(cmd) miinc

e(miinc) se or me

e(ic) contents of the ic() option

e(reg) contents of the reg() option

e(test) test type for coefficients (affected by ttest)

e(set#) independent variables or parameters included in set#

e(all) contents of the all() option or all parameters estimated but not in equations or sets()

matrices

e(b) model-averaged coefficient vector

e(V) model-averaged estimator variance-covariance matrix

e(pip) posterior inclusion probability vector

e(best_mods) best models matrix

functions

e(sample) marks estimation sample

References

Cui, J. (2007). QIC program and model selection in GEE analyses. Stata journal, 7(2), 209-220.

Burnham, K. P., & Anderson, D. R. (2002). Model selection and multimodel inference: a practical information-theoretic approach. Springer.

Burnham, K. P., & Anderson, D. R. (2004). Multimodel inference understanding AIC and BIC in model selection. Sociological methods & research, 33(2), 261-304.

Scott, A. (August, 2013). Information criteria under complex sampling. Proceedings 59th ISI World Statistics Congress, Hong Kong (Session STS075). Rerieved from http://2013.isiproceedings.org/Files/STS075-P3-S.pdf.

Author

Joseph N. Luchman

Behavioral Statistics Lead

Fors Marsh Group LLC

Arlington, VA

jluchman@forsmarshgroup.com