*! 28sept2011
help cddens

Title

Density Estimation under Measurement Error using Auxiliary Information

Syntax

cddens
observed
[predictors]
[if] [in]
[weight]
[, options]

Basic Syntax
cdf estimates the cdf instead of the pdf

cdfvar(varlist) adds the cdf of each variable in varlist to the plot if cdf is specified

parameters(matname) name of vector that contains the parameters of the conditional distribution. Required unless model(auto) is used.

model(string) parametric family of the conditional distribution used for measurement error and optionally non-response

mvars(varlist) list of variables used to predict the true value if the observed value is missing. Default is constants only.

masspoint needs to be specified if conditional density contains a mass point

kdensity_options options passed to kdensity/line

draws(#) number of draws per observation used to integrate over conditional density. Default is 40.

save(filename [, varlist]) save draws and variables varlist using filename

tname(varname) name of true, unobserved variable (used in density plot)

ksmirnov(varname) Kolomogorov-Smirnov test whether simulated draws come from same distribution as varname

ckolmogorov(varname [,#]) Conditional Kolmogorov test whether varname comes from the conditional distribution specified

Advanced Syntax
predictors(string) comma separated string that contains the number of predictors (incl. constant) for each parameter of the conditional density.
Needs to be specified if parameters() does not contain equation names.

mixfunc(pointer) pointers to the functions used in the conditional density for measurement error.

mixpar(string) vector specifying the number of input arguments for the functions in the measurement error part of the likelihood function.

mixtransf(string) specifies parameter transformations for measurement error that need to be implemented before drawing from the conditional density.

wgttrfunc(pointer) pointer to function used to transform weights of mixture components for measurement error

mpfunc(pointer) pointer to the function that transforms the mass point into a probability.

nrfunc(pointer) pointers to the functions used in the conditional density for missing values

nrpar(string) vector specifying the number of input arguments for the functions in the non-response part of the likelihood function.

nrtransf(string) specifies parameter transformations for missing values that need to be implemented before drawing from the conditional density.

nrwgttrfunc(pointer) pointer to function used to transform weights of mixture components for missing values

fweights, aweights and iweights are allowed; see weight.

Description

cddens consistently estimates the density of a variable that is only observed with error using parameter estimates of the conditional density of the true
value given the observed value. Please see the papers in the References below for more details on the method and how it can be used.

Options

The basic syntax can be used if the conditional distribution comes from (mixtures of) several common parametric distributions.
The advanced syntax needs to be used if the parameter vector passed to cddens characterizes other densities.

Basic Syntax

cdf or cdf(varlist) makes the program estimate the cdf of the mismeasured variable instead of the pdf, adding any additional variables in varlist to the plot. If either is specified,
additional options are passed to line instead of kdensity.

parameters(matname) needs to contain the name of the vector in memory in which the parameter estimates of the conditional distribution are stored. The ordering of the parameters in the vector needs
to be as follows:

[mass point parameters] [non-response parameters] measurement error parameters

If the model for non-response or measurement error contains mixtures of densities, it is assumed that the vector does not contain a weight for the first component
(as it is implicitly denfined by the other weights) and the weight of each component is located after the parameters of the component, i.e.:

param for component 1, param for component 2, weight for component 2, param for component 3, weight for component 3, ...

If the vector has equation names, cddens can infer predictors() from these names, so it does not have to be specified. parameters()
is required unless model(auto) is used.

model(string) can be used to specify several parametric models for measurement error and missing values. Can contain one or two of the following options, where the first specifies the model for
measurement error and the second specifies the model for missing values. The options are:

auto: obtains all information from e(). To be used after conddens, does not require second entry for missing values.

normal: (mixture of) normal densities

ltrnorm, rtrnorm, trnorm: truncated normal distributions (left, right and both sides).

mixltrnorm, mixrtrnorm, mixtrnorm: 2 component mixtures of truncated normal distributions (left, right and both sides).

exp: (mixture of) exponential densities with location parameter

weibull: (mixture of) weibull densities with location parameter

t: (mixture of) generalized t-distribution(s) with scale and location parameter, df is constrained to be >=1

mixweit: mixture of a weibull distribution with location parameter and a generalized t-distribution with scale and location parameter

Note: model() does not set which parameters are predicted by the data and which are constants. If this information cannot be obtained from the equation and parameter names of parameters(), it
needs to be entered manually using predictors(). See predictors() for more detail. model() overrides all advanced options required by the specified model except for
predictors. Combining it with advanced options that are not required by the model (such as parameter transformations) should be done with caution.

mvars(varlist) can be used to specify a list of variables that predicts the true value if the observed value is missing. The default is to use constants only for the parameters in the model for missing values.

masspoint needs to be specified if the conditional density contains a mass point at true=observed to inform program that the parameter vector contains parameters for a masspoint.

display_options if any options not described here are specified, they will be passed to kdensity (or line if cdf is specified). See kdensity and line for more details on which options are allowed.
One should be careful with options that depend on the sample size, since estimation is run on _N*d observations, while the actual sample size is _N (d being the number of simulated draws).

draws(#) cddens uses random draws from the conditional density to integrate it out. draws() gives the number of draws per observation, the default is 40.

save(filename [, varlist]) saves the random draws used to simulate the integral in filename.dta along with the variables in varlist and e(sample). The draws are saved as one column vector, repeating the original
dataset as many times as specified in draws(): if the original dataset contains n observations, the saved dataset will contain d*n observations (even if not all n observations are used). However, except for
the simulated variable, all observations after the first n will be missing (they can be filled by repeating the first n observations d times).

tname(varname) can be used to specify a variable name that will be used in the output for the latent true variable.

ksmirnov(varname) tests whether varname is a sample from the latent true variable by performing a two sample Kolmogorov-Smirnov test whether varname and the simulated sample from the true distribution
are drawn from the same distribution. If weights are specified, ksmirnov2.ado needs to be installed or the test will be performed without weights.

ckolmogorov(varname [,#]) performs a conditional Kolmogorov test (Andrews 1997) whether varname is a sample from the conditional distribution specified. The test is performed separately for observations
subject to measurement error and observations subject to non-response, so it produces two test statistics if a model for non-response is specified. If the optional # is specified, # bootstrap iterations
will be used to find the p-value(s) of the test statistic(s). Note that the p-value is calculated under the assumption of iid observations. The test currently only works with the basic syntax, to make it work with
other models, you will need to add the conditional cdf to the .ado file.

Advanced Syntax

predictors(string) specifies how many elements of the vector in parameters() belong to each parameter of the conditional distribution. E.g. if the first parameter of the conditional distribution is
predicted by 4 variables and a constant, the first entry of predictors() should be 5, it should be 1 if the first parameter is a constant. Consequently, the sum of the entries of predictors() should be
the same as the length of the parameter vector in parameters(). Required if the parameter vector does not contain equation names that allow cddens to infer this information.

mixfunc(pointer) should contain pointers to the functions used in the measurement error part of the conditional density (e.g. mixffunc(&rgent() &rgent() for a mixture of two generalized t-distributions.
cddens will use these pointers to make draws from the conditional density in order to integrate out the measurement error. Note that Stata does not allow pointers to functions that are built-in, so one has
to define a separate function for these. See Pointers to Functions for more details. The functions specified in mixfunc() are reused if the number specified does not match the length
of the parameter vector.

mixpar(string) String of numbers specifying the number of input arguments for each function in mixfunc(). It is assumed that each entry of mixpar() except for the first one is followed by a weight
in the parameter vector in parameters(). mixpar will be repeated to make the number of arguments match the length of the parameter vector if possible. For example, the normal density requires two inputs (mean and
variance), so if the conditional distribution is normal, use mixpar(2). A mixture of two normals can be specified by mixpar(2,2) or mixpar(2), a mixture of a normal and a 3-parameter t distribution
requires mixpar(2,3).

mixtransf(string) mixtransf() can be used to transform parameters before they are used in the functions in mixfunc(). E.g. if the conditional distribution is normal with mean log(Xb), a
log transformation for the first parameter needs to be specified. Each transformation requires three entries in mixtransf(): the first gives the position of the function in mixfunc(), the second
the position of the parameter in that function and the third contains either a pointer to the transformation function or a number (which is added to the parameter). E.g. to take the inverse logit of the second
argument of the first function in mixfunc(), use

mixtransf(1,2,&invlogit())

To implement multiple transformations, simply add more entries to the list, e.g. to additionally add 5.13 to the first parameter of the second function in mixfunc(), use

mixtransf(1,2,&invlogit(),2,1,5.13)

The same parameter can be transformed multiple times with transformations implemented in the order they appear in mixtransf(). Entries are repeated if entries in mixpar() are repeated.

wgttrfunc(pointer) Can be specified if the weights of the mixture distribution for measurement error need to be transformed. Note that the same transformation is applied to all weights, so contrary to
mixtransf(string), wgttrfunc() should only contain the pointer to the function that transforms the weights (e.g. wgttrfunc(&invlogit()).

mpfunc(pointer) specifies pointer to the function that transforms the mass point into a probability, i.e. F() such that Pr(true=observed)=F(Xb). The default is the Probit function (normal cdf).

nrfunc(pointer) Analog of mixfunc() for missing values, see there for details. Contrary to mixfunc(), nrfunc() is not repeated, so it always needs to contain as many entries as there
are components in the mixture for non-response.

nrpar(string) Analog of mixpar() for missing values, see there for details. Contrary to mixpar(), nrpar() is not repeated, so it always needs to contain as many entries as there
are components in the mixture for non-response.

nrtransf(string) Analog of mixtransf() for missing values, see there for details. Like with the options above, entries are not repeated, so all transformations have to be specified.

nrwgttrfunc(pointer) Analog of wgttrfunc() for missing values, see there for details.

Remarks

Required options and defaults

Usually, you will need to specify parameters() and model(). If the vector in parameters() does not have equation names, you also need to provide predictors().
If you have just estimated the conditional density and the results are still stored in e(), model(auto) can be specified by itself. You can leave out model() if
you specify (at a minimum) mixfunc() and mixpar(). Things will usually default to a normal or a mixture of normals if you fail to specify required options. I have not
worked with the advanced syntax a lot so far and there are many ways to specify it. So if you use it, please carefully check that what you get is right and let me know in case anything goes wrong.

Variables predicting parameters

cddens uses all variables in observed predictors as predictors for the mass point and all parameters for which predict!=1 of the measurement error model. Similarly, all
variables in mvars() are used as predictors for all parameters of the non-response model that are not constants. If some variables are to be excluded from some parameter equations, the corresponding entry
in the parameter vector should be a zero, it should not be excluded from the vector.

Pointers to Functions

In order to integrate over the conditional distribution, cddens produces random draws from the conditional distribution for each observation. To do so, it needs to be supplied with pointers to the mata
functions that generate random draws from the (components of) the conditional density. A pointer to a function is the mata name of the function prefixed with "&" and followed by "()", i.e. &invlogit() is a
pointer to matas "invlogit" function. See Pointers for more details. These pointers are automatically set with model(), but need to be supplied through mixfunc() and nrfunc()
when using the advanced syntax. The functions should take as many inputs as specified in mixpar() and nrpar() and need to return vectors of random draws from the density that have the same length
as the input vectors. Since Stata does not allow pointers to built in functions (such as rt(), rnormal, etc.), these have to be replaced by user defined functions. E.g. to use a standard t-distribution,
first define the function as follows in mata:
function mrt(df) return(rt(df))
and then run cddens with option mixfunc(&mrt()). The same applies to functions used for parameter transformations. The following random number functions are already defined in cddens because the
original functions are built-in functions:

mrnormal(a,b): produces random draws from a normal density with mean a and sd b

rexploc(a,b): produces random draws from an exponential density with location parameter a and scale parameter b.

rweibloc(a,b,c): produces random draws from a weibull density with location parameter a, shape parameter b and scale parameter c

rgent(a,b,c): produces random draws from a generalized t-distribution with location parameter a, scale parameter b and degrees of freedom c (constrained to be >=1)

The following transformation functions are also already defined:

mabs(a): returns the absolute value of a

mneg(a): returns the negative value of a (-1*abs(a)).

mexp(a): returns e^a

Other

If you find any mistakes or have any suggestions for improvements, please send me an email to mittag@uchicago.edu. Feel free to use, change or mutilate this program
for private purpose, but please don't steal it, give due credit.

Examples

Basic Syntax

Predicting a variable using obs_inc race age gender and parameters of a mixture of two normal distributions with mass point stored in matrix "par"

. cddens obs_inc race age gender, parameters(par) model(mixnorm) mass

Predicting a variable using obs_inc race age gender and parameters of a mixture of two t-distributions with mass point for measurement error and a mixture of two normal distributions for
non-response stored in matrix "par"

. cddens obs_inc race age gender, parameters(par) model(mixt mixnorm) mvars(race age gender)

Saved results

cddens saves all results of kdensity and summarize, d in e(), see kdensity and summarize for details. If ksmirnov() is specified, it will also save the results
of ksmirnov, see ksmirnov. If ckolmogorov is used, it will additionally save

Scalars

e(ck) Test Statistic for the conditional density of measurement error

e(p_ck) bootstrapped p-value of the test statistic for measurement error (optional)

e(cknr) Test Statistic for the conditional density of non response

e(p_cknr) bootstrapped p-value of the test statistic for non response (optional)

e(ck_bsi) number of bootstrap iterations used to calculate the p-values (optional)

References

Andrews,D.W.K. 1997. A Conditional Kolmogorov Test. Econometrica. 65(5): p. 1097-1128.

Mittag, N. 2013. "A Method of Correcting for Misreporting Applied to the Food Stamp Program." Unpublished Manuscript

Mittag, N. 2013. "Imputations: Benefits, Risks and a Method for Missing Data." Unpublished Manuscript

Author

Nikolas Mittag, University of Chicago
mittag@uchicago.edu