*! version 1.22 24jul2014

Title

fastxtile -- Drop in replacement for xtile, which runs significantly faster on large datasets

Syntax

Create variable containing quantile categories

fastxtile
newvar = exp
[if] [in]
[weight]
[, options]

options

Main
nquantiles(#) number of quantiles; default is
nquantiles(2)

cutpoints(varname) use values of varname as cutpoints

altdef use alternative formula for calculating
percentiles

Additions in fastxtile
cutvalues(numlist) use values of numlist as cutpoints; numlist must be strictly ascending

randvar(varname) use varname to sample observations when computing quantile boundaries

randcut(#) upper bound on randvar() used to cut the sample; default is randcut(1)

randn(#) number of observations to sample when computing quantile boundaries

aweights, fweights, and pweights are allowed
(see [U] 11.1.6 weight), except when altdef, cutpoints(varname) or cutvalues(numlist)
are specified, in which case no weights are allowed.

Description

fastxtile is a drop in replacement for [D] xtile. It has the same syntax and produces identical results, but has been optimized to be more computationally efficient. The
difference in running time is substantial in large datasets.

It also supports computing the quantile boundaries using a random sample of the data.
This further increases the speed, but generates approximate quantiles due to sampling error.

Options

Main

nquantiles(#) specifies the number of quantiles.
It computes percentiles corresponding to percentages 100*k/m
for k=1, 2, ..., m-1, where m=#. For example, nquantiles(10)
requests that the 10th, 20th, ..., 90th percentiles be computed. The default
is nquantiles(2); i.e., the median is computed.

altdef uses an alternative formula for calculating percentiles.
The default method is to invert the empirical distribution function by using
averages, where the function is flat (the default is the same method used by
summarize; see [R] summarize).
The alternative formula uses an interpolation method. Weights cannot be used
when altdef is specified.

cutpoints(varname) requests that fastxtile
use the values of varname, rather than quantiles, as cutpoints for the
categories. All values of varname are used, regardless of any if
or in restriction. This option cannot be combined with any other options.

Additions in fastxtile

cutvalues(numlist) specifies a list of values to be used as
cutpoints for the categories. The list must be ascending and contain no
repeated values. This option cannot be combined with any other options.

randvar(varname) requests that varname be used to select a
sample of observations when computing the quantile boundaries. Sampling increases
the speed of fastxtile, but generates approximate quantiles due to sampling error.
It is possible to omit this option and still perform random sampling from U[0,1]
as described below in randcut(#) and randn(#).

randcut(#) specifies the upper bound on the variable contained
in randvar(varname). Quantile boundaries are approximated using observations for which
randvar() <= #. If no variable is specified in randvar(),
a standard uniform random variable is generated. The default is randcut(1).
This option cannot be combined with randn(#).

randn(#) specifies an approximate number of observations to sample when
computing the quantile boundaries. Quantile boundaries are approximated using observations
for which a uniform random variable is <= #/N. The exact number of observations
sampled may therefore differ from #, but it equals # in expectation. When this option is
combined with randvar(varname), varname ought to be distributed U[0,1].
Otherwise, a standard uniform random variable is generated. This option cannot be combined
with randcut(#).

Saved results

fastxtile saves the following in r():

Scalars

r(r#) value of #-requested percentile

r(N) number of observations

r(n) number of sampled observations when computing quantile boundaries

Author

Michael Stepner

stepner@mit.edu

Acknowledgements

The fastxtile code and documentation are based on the built-in xtile command.

Raj Chetty provided the impetus for this program, and the idea of using random sampling to generate
approximate quantiles quickly in large datasets.

Laszlo Sandor suggested the option randn(), pointing out that the expected error in
the size of the bins is proportional to the sample size, not the sampling rate.

Also see

Manual: [D] pctile

Help: [D] pctile