*! 05jun2011
help conddens

Title

Estimation of a conditional density to correct for measurement error and missing values

Syntax

conddens
true
[observed]
[predictors]
[if] [in]
[weight]
[, options]

Basic Syntax
model(string) parametric family used for measurement error and optionally non-response

mvars(varlist) list of variables used to predict the true value if the observed value is missing. Default is constants only.

masspoint adds a mass point at true=observed to the specified model

modelopts(string) options passed to ml model

maxopts(string) options passed to ml maximize

init(string) starting values passed to ml init

search(string) controls ml search or switches it off

eqnames(string) equation names for parameter equations

cmplot runs a nonparametric regression of true on the predicted conditional mean.

cmpopts(string) passes options string to lpoly if cmplot is specified.

macrokeep prevents program from dropping macros

Advanced Syntax
likprog(name) name of program that evaluates the likelihood function for ml. Default is condmix

predict(string) string of 0s and 1s specifying whether parameters are functions of the data

mpfunc(string) name of the link function for the mass point, if different from Probit

mixpar(string) vector specifying the number of input arguments for the functions in the measurement error part of the likelihood function.

mixtransf(string) specifies parameter transformations of constants for measurement error in likprog

mixlik(string) likelihood function for measurement error passed to condmix

mixargs(string) parameter names for measurement error used in condmix

mixfunc(string) vector of pointers to functions that generate random draws from the mixture components

wgttrfunc(pointer) pointer to function used to transform weights of mixture components for measurement error in condmix

nrfunc(pointer) pointers to the functions used in the conditional density for missing values

nrpar(string) vector specifying the number of input arguments for the functions in the non-response part of the likelihood function.

nrtransf(string) specifies parameter transformations of constants for missing values in likprog

nrlik(string) likelihood function for missing values passed to condmix

nrargs(string) parameter names for missing values used in condmix

nrwgttrfunc(pointer) pointer to function used to transform weights of mixture components for missing values in condmix

fweights, aweights, iweights, and pweights are allowed; see weight. Technically, the program should work with bootstrap and svy,
but I have not tested the results beyond very basic cases.

Description

conddens estimates flexible parametric densities of a true variable conditional on its observed value (which can be missin) and other predictors. The command can be used
on a validation dataset and the resulting parameter estimates can be passed to a program like cddens in order to obtain consistent estimates if the true values are not available.
Please see the papers in the References below for more details on the method and how it can be used.

Options

The basic syntax allows estimation of several parametric models. The advanced syntax can be used to override some default features of these models or to specify more complex models.

Basic Syntax

model(string) can be used to specify several parametric models for measurement error and missing values. Can contain one or two of the following options, where the first specifies the model for
measurement error and the second specifies the model for missing values. The options are:

normal: normal density with mean as a function of the data

ltrnorm, rtrnorm, trnorm: truncated normal distributions (left, right and both sides).

mixnorm: mixture of two normal densities with means as functions of the data

mixltrnorm, mixrtrnorm, mixtrnorm: 2 component mixtures of truncated normal distributions (left, right and both sides).

exp: exponential density with location parameter (constrained to be >=min(true)) and lambda as a function of the data

mixexp: mixture of two exponential densities with location parameters (one constrained to be >=min(true)) and lambda as functions of the data

weibull: weibull density with location parameter (constrained to be >=min(true)), scale parameter is a function of the data

mixweibull mixture of two weibull densities with location parameters (one constrained to be >=min(true)), scale parameters are a function of the data

t: generalized t-distribution with scale and location parameter, mean is a function of the data, df is constrained to be >=1

mixt: mixture of two generalized t-distributions with scale and location parameters, means are a function of the data, df is constrained to be >=1

mixweit: mixture of a weibull distribution with location parameter and a generalized t-distribution with scale and location parameter

Notes:

1. model() can only be specified if the likelihood program is condmix.

2. predict() can be used with model() to change which parameters are predicted by the data. eqnames() can be used to change the default equation names, model() overrides all other advanced options.

mvars(varlist) can be used to specify a list of variables that predicts the true value if the observed value is missing. The default is to use constants only for the parameters in the model for missing values.

masspoint allows the conditional density to have a masspoint where the true value is equal to the observed value (correct reporting). The masspoint is a function of the data using
a Probit transformation, i.e. P(true=obs)=normal(Xb), where b is estimated from the data.

modelopts(string) can be used to pass any other options to ml model. Options should be entered just as they would be entered after the comma in the ml model statement.

maxopts(string) can be used to pass any other options to ml maximize. Options should be entered just as they would be entered after the comma in the ml maximize statement.

init(string) can be used to pass starting values to ml using ml init. Options should be entered as they would be entered after the comma in the ml init statement. Using a format
different from equation:par=value should work, but may cause problems with the default initial values. If the starting values are stored as a matrix in memory, use init(matrix matname, initoptions)
Using init(random) switches off the default starting values and lets ml search determine them randomly. See Starting Values for more details.

search(string)string can be a number to run ml search that many times to find starting values or "off" to switch ml search off.

eqnames(string) can be used to specify equation names in ml and should contain a string with one entry for each equation (specifying only some names may cause problems).

cmplot performs a misspecification test of the conditional mean function against semiparametric alternatives by running a non-parametric (local polynomial) regression of the true variable on its estimated
conditional mean. Horowitz and Haerdle (1994) describe this idea and derive a formal test using Kernel regression, cmplot only plots the nonparametric fit with confidence intervals against a 45-degree-line.
Departures from the 45-degree-line indicate misspecification of the conditional mean. Note that the nonparametric regression uses the lpoly command (see lpoly for details), so it only accepts
fweights and aweights, ignoring other weights if specified. cmpopts(str) passes str as options to lpoly. Note further that local polynomial regressions tend to show erratic behaviour in the tails of the distribution. The option currently only works
with the basic syntax.

macrokeep prevents conddens from dropping the internal macros at the end. Can be useful if you want to run postestimation commands.

Advanced Syntax

likprog(name) makes ml use program name instead of the default condmix to evaluate the likelihood function. See the documentation for ml to see the requirements such a program has to
meet.

predict(string) specifies which parameters of the conditional distribution are functions of the data and which are constants. Should be a string of 0s and 1s of the same length as the parameter vector of the
conditional distribution. A parameter is predicted by the variables in varlist (mvars for missing values) if the corresponding entry is 1 and is set to a constant if it is zero. See
the remark below (Variables Predicting Parameters) for further details on which variables predict parameters. The ordering of the parameters is

[masspoint parameters] [parameters for missing values] parameters for measurement error.

If the density for missing values or measurement error is a mixture, it is assumed that the first component contains no weight (its weight is 1-sum of other weights) and the weight is always the last parameter of each component. I.e.:

parameters for first component, [parameters for second component, weight of second component, parameters for third component, weigth of third component,...]

mpfunc(string) Can be used to use a different link function for the mass point, e.g. the inverse logit function. Needs to be specified as the name of the function, not as a pointer, e.g. mpfunc(invlogit).

mixpar(string) String of numbers specifying the number of input arguments each component of the mixture for measurement error requires. It is assumed that each entry of mixpar except for the first one is
followed by a weight. mixpar will be repeated to make the number of arguments match the length of the parameter vector returned by ml if possible. For example, the normal density requires two inputs (mean and
variance), so if the conditional distribution is normal, use mixpar(2). A mixture of two normals can be specified by mixpar(2,2) or mixpar(2), a mixture of a normal and a 3-parameter t distribution
requires mixpar(2,3).

mixtransf(string) In many cases, it is convenient to use parameter transformations in the likelihood functions, e.g. to estimate the log of the variance of a normal distribution in order to make sure
the variance is positive. In order for other programs to use the output of conddens, they have to be supplied with information to undo these transformations and obtain the real parameter. mixtransf() can
be used to implement this. Each transformation requires three entries in mixtransf(): the first gives the position of the function in mixfunc(), the second the position of the parameter in that function
and the third contains either a pointer to the transformation function or a number (which is added to the parameter). E.g. to take the inverse logit of the second argument of the first function in mixfunc(), use

mixtransf(1,2,&invlogit())

To implement multiple transformations, simply add more entries to the list, e.g. to additionally add 5.13 to the first parameter of the second function in mixfunc(), use

mixtransf(1,2,&invlogit(),2,1,5.13)

The same parameter can be transformed multiple times with transformations implemented in the order they appear in mixtransf(). Entries are repeated if entries in mixpar() are repeated.
For parameters that are constants, mixtransf() will attempt to transform the parameter before output is displayed and stored. If a parameter is transformed already, this is indicated by a "t"
in the first column of the output table. The untransformed parameters and the original variance matrix are saved in e(b_o) and e(V_o). Entries for which the parameter could not be transformed are
saved in e(mixtransf) and can be used by other programs.

mixlik(string) Can be used to specify the measurement error part of the likelihood function in condmix manually. Requires the arguments to be used to be specified in mixargs(). These arguments
have to appear as locals in mixlik(), all locals (and globals) in mixlik() have to be preceeded by "\" (e.g. \`par1') to make sure they are not evaluated before being passed to the program. See
ml for further details on how to specify likelihood functions.

mixargs(string) Needed if the measurement error part of the likelihood function is specified manually using mixlik(). Should contain a list of the arguments used in the likelihood function. E.g. if
the likelihood function is maximized with respect to parameters callen "mean" and "variance", use mixargs(mean variance).

mixfunc(string) Can be used to specify pointers to functions that take random draws from the mixture components of the conditional density if the advanced syntax is used. Currently, this is only passed
on to other programs in a macro and not actually needed unless you need it stored in a macro.

wgttrfunc(pointer) It is often convenient to estimate a transformation of the weights in order to make sure they stay within [0,1]. wgttrfunc() is needed to undo this transformation or to enable other
programs to do this (if weights are not constants). Note that the same transformation is applied to all weights, so contrary to mixtransf(string), wgttrfunc() should only contain the pointer to the
function that transforms the weights (e.g. wgttrfunc(&invlogit()).

nrfunc(pointer) Analog of mixfunc() for missing values, see there for details. Contrary to mixfunc(), nrfunc() is not repeated, so it always needs to contain as many entries as there
are components in the mixture for non-response.

nrpar(string) Analog of mixpar() for missing values, see there for details. Contrary to mixpar(), nrpar() is not repeated, so it always needs to contain as many entries as there
are components in the mixture for non-response.

nrtransf(string) Analog of mixtransf() for missing values, see there for details. Like with the options above, entries are not repeated, so all transformations have to be specified.

nrlik(string) Analog of mixlik() for missing values, see there for details. Allows manual specification of the likelihood part for missing values if condmix is used as the likelihood program.

nrargs(string) Analog of mixargs() for missing values, see there for details.

nrwgttrfunc(pointer) Analog of wgttrfunc() for missing values, see there for details.

Remarks

Required options and defaults

Usually, you will only need to specify model(), which overrides all other options except for predict and eqnames.
predict is required if any part of the model is changed from the default in model. You can leave out model() if
you specify the model with the advanced syntax. Things will usually default to a normal or a mixture of normals if you fail to
specify required options. I have not worked with the advanced syntax a lot so far and there are many ways to specify it. So if you
use it, please carefully check that what you get is right and let me know in case anything goes wrong.

Variables predicting parameters

conddens uses all variables except for the first variable (true value) as predictors for the mass point and all parameters for which predict==1 of the measurement error model. Similarly, all
variables in mvars() are used as predictors for all parameters of the non-response model that are not constants. If some variables are to be excluded from some parameter equations, this can
be done by defining constraints that set them to zero and passing them to ml using modelopts(). See ml for more information on how to do this.

Transformations

For computational reasons, it is often desirable to estimate transformations of some parameters (such as the log of a variance to constrain it to be positive). Such transformations need to be undone
in order to use the conditional density to correct other models and can be specified using mixtransf(), nrtransf(), wgttrfunc() and nrwgttrfunc(). Whenever possible, conddens
implements these transformations for the parameter, the gradient and its enttries in the covariance matrix (using the delta method). No confidence intervals, t-statistics and p-values are displayed for
transformed parameters and they are marked by a "t" at the beginning of the line in the output. The untransformed parameter vector and covariance matrix are saved in e(b_o) and e(V_o). If conddens cannot
perform a transformation that is specified, it saves it in e(mixtransf), e(nrtransf), e(wgttrfunc) or e(nrwgttrfunc).

Starting Values

conddens chooses starting values for (mixtures of) t and normal distributions by running regressions, but for all other distributions currently only sets the starting values of the weights to 0.5 if a
mixture is specified using model(). ml does a random search, but a lack of decent starting values may prevent or slow down convergence, so if possible, it is a good idea to supply initial values
using init(). If convergence fails, the values of the last iteration can be displayed using ml report. Using starting values from a regression for the components of a mixture may not be a good idea
if the components are dissimilar. In such cases starting values should be provided manually (overriding the defaults) or init(random) can be used to have ml search pick random starting values.

Other

If you find any mistakes or have any suggestions for improvements, please send me an email to mittag@uchicago.edu. Feel free to use, change or mutilate this program
for private purpose, but please don't steal it, give due credit.

Examples

Basic Syntax

Mixture of two normal distributions with mass point

. conddens true_inc obs_inc race age gender, model(mixnorm) mass

Mixture of two t-distributions for measurement error and two normal distributions for missing values

. conddens true_inc obs_inc race age gender, model(mixt mixnorm) mvars(race age gender)

Saved results

conddens saves all results of ml in e(), see ml and maximize for details. If cmplot is specified, it also saves the results of lpoly in e(), see lpoly for details.
In addition to these results, conddens saves the following in e():

Scalars

e(min) smallest value of the observed variable used to estimate the conditional density.

e(max) largest value of the observed variable used to estimate the conditional density.

Macros

e(cmd) conddens

e(obs) name of variable that contains the observed values

e(pred) names of predictor variables for measurement error

e(mvars) names of predictor variables for non-response

e(mixfunc) vector of pointers to functions used in measurement error part of likelihood

e(mixtransf) string of parameter transformations for measurement error parameters that have not yet been transformed. See Transformations for further details.

e(wgttrfunc) pointer to function that implements the transformation of the weights for measurement error if they have not yet been transformed.

e(nrfunc) vector of pointers to functions used in non-response part of likelihood

e(nrtransf) string of parameter transformations for non-response parameters if they have not yet been transformed. See Transformations for further details.

e(wgttrfunc) pointer to function that implements the transformation of the weights for non-response if they have not yet been transformed.

e(mpfunc) pointer to function used to transform mass point into probability.

Matrices

e(b) coefficient vector after parameter transformations have been implemented

e(b_o) original coefficient vector returned by ml

e(V) variance-covariance matrix of the estimators after parameter transformations have been implemented

e(V_o) original variance-covariance matrix of the estimators returned by ml

e(gradient) gradient vector after parameter transformations have been implemented

e(predict) Entries of this vector equal 0 if the corresponding parameter of the conditional density is a constant and 1 if it is a function of the variables in varlist or mvars()

e(predictors) Vector specifying how many parameters of the output vector belong to each equation (i.e. predict each parameter). Same length as e(predict), but
gives the number of parameters rather than 0/1.

e(mixpar) Vector of the number of parameters for the functions of the measurement error part of the likelihood.

e(nrpar) Vector of the number of parameters for the functions of the non-response part of the likelihood.

References

Horowitz, J.L. and W. Haerdle 1994. "Testing a Parametric Model against a Semiparametric Alternative." Econometric Theory. 10: p. 821-848.

Mittag, N. 2013. "A Method of Correcting for Misreporting Applied to the Food Stamp Program." Unpublished Manuscript

Mittag, N. 2013. "Imputations: Benefits, Risks and a Method for Missing Data." Unpublished Manuscript

Author

Nikolas Mittag, University of Chicago
mittag@uchicago.edu