*! version 1.1.3 8apr2021

outdetect -- Outlier detection and diagnostics for welfare analysts

Syntax

outdetect varname [if] [in]
[weight]
[, options]

options

Main
normalize(normtype) specify the transformation for normalizing varname; default is Yeo and Johnson (2000)

bestnormalize use the best fitting transformation for normalizing varname

normvar(varname) create a new variable containing the normalized variable

zscore(stat1 stat2) define the z-score of the normalized variable; default is stat1 = median,
stat2 = Q-statistic (Rousseeuw and Croux, 1993)

alpha(#) specify the threshold of the outlier detection region; default is 3

outliers(bottom|top|both) specify whether outliers are to be flagged at the bottom, top, or on both sides of the distribution of varname

nonegative exclude negative values of varname from all calculations

nozero exclude zero values of varname from all calculations

nogenerate do not create _out variable

replace replace existing _out variable

reweight create a new variable containing the post-detection adjusted weights. Only if weights or svyset are specified

madfactor(#) specify Fisher consistency factor for median absolute deviation; default is 1.4826

sfactor(#) specify Fisher consistency factor for S-statistic; default is 1.1926

qfactor(#) specify Fisher consistency factor for Q-statistic; default is 2.2219

small apply small samples correction for MAD, S and Q statistics

Reporting
sformat(%fmt) specify format for summary statistics panel

pline(# | varname) specify a poverty line and report poverty estimates

excel(filename [, replace]) export output in Excel

Graphics
graph(gtype) produce diagnostic plots

pweights are allowed; see weights.

svyset can be used to designate variables containing information about the survey design, such as the sampling units and probability weights.

Description

outdetect identifies extreme values, either "too small" or "too large" observations, in the distribution of varname. We shall call these observations bottom outliers and top outliers, respectively
(small values being at the bottom of the distribution of varname, and large values being at the top).

Users may exploit svyset to specify the survey design of the data before using outdetect.
By default, outdetect creates a new variable, _out, containing numeric codes that flag outliers of varname:

Numeric
code Description

0	observation is not an outlier
1 observation is a bottom outlier ("too small")
2 observation is a top outlier ("too large")

The output of outdetect reports "Raw" statistics (computed using varname), as well as "Trimmed" statistics (computed using just those observations of varname that are not flagged as outliers).

The procedure by which outliers are detected is described in Belotti et al. (2021), and involves two steps.
First, the distribution of the target
variable (varname) is transformed to approach a standard normal distribution.
To do so, varname is normalized (a transformation is applied so that its distribution approaches a Normal), then standardized (a z-score of the transformed variable is computed).
Second, a threshold is applied to the transformed variable, to set the bounds of an outlier detection region (typically corresponding to the tails of the transformed distribution).
The threshold is set at a conventional value (e.g. 3).

In formulas:

target variable (varname) = v

normalized variable = x = t(v)

where t is a normalizing transformation.

z-score of x = z = (x - stat1)/stat2

where stat1 is a measure of location and stat2 is a measure of scale.

alpha = conventional threshold

An observation of v is flagged as an outlier if:

|z| > alpha (both bottom and top outliers), or

z > alpha (top outlier), or

z < -alpha (bottom outlier).

Options

Main

normalize(normtype) specifies the method for transforming varname into a distribution that approaches a Normal distribution. normtype may be selected among the following transformations:

normalize(yj) applies Yeo and Johnson (2000) (default);

normalize(asinh) applies the inverse hyperbolic sine (Friedline et al. 2014).

normalize(bcox) applies the Box-Cox transform (Box and Cox 1964);

normalize(ln) applies the natural logarithm, i.e. ln(x);

normalize(log10) applies log10(x + a) with a = max(0, -(min(x) - 0.0001));

normalize(log) applies log(x + a) with a = max(0, -(min(x) - 0.0001));

normalize(sqrt) applies the square root;

normalize(none) applies no transformation (varname used as is).

bestnormalize selects the best transformation according to the value of the Pearson P statistic divided by its degrees of freedom (df) (Snedecor and Cochran, 1989).
This ratio converges to 1 when the data approaches a Gaussian distribution.
Therefore, the Pearson/df ratio can be interpreted as a measure of how close a distribution is to normality, and used to rank transformations according to how successful they are in normalizing the data.
When this option is specified, outdetect stores the Pearson/df ratio corresponding to the best normalizing transformation.

normvar(varname) creates a new variable corresponding to the normalization specified via normalize() or the best normalization identified by bestnormalize.

zscore(stat1 stat2) specifies the z-score of the normalized variable. If x is the normalized variable, the z-score is defined as z = (x - stat1)/stat2.

stat1 can be chosen among the following:

mean, mean of x;

median, median of x (default);

stat2 can be chosen among the following:

std, standard deviation of x;

mad, median absolute deviation;

iqr, interquartile range;

s, S-statistic (Rousseeuw and Croux, 1993);

q, Q-statistic (Rousseeuw and Croux, 1993) (default);

alpha(#) specifies the threshold of the outlier detection region, which is defined with reference to the distribution of the z-score. The default is 3, but conventional values may range between 2 and 4, depending on the context.

outliers(bottom|top|both) specifies whether outliers are to be flagged at one or both sides of the distribution of varname:

bottom, only flags bottom ("too small") outliers;

top, only flags top ("too large") outliers;

both, flags both bottom and top outliers (default).

nonegative excludes negative values of varname from the detection routine, the computation of summary statistics, and all other calculations.

nozero excludes zeros of varname from the detection routine, the computation of summary statistics, and all other calculations.

nogenerate specifies that variable _out (which flags outliers of varname) not be created.

replace replaces any existing variable named _out with the new _out variable created by issuing outdetect.

reweight creates a new variable containing the post-detection adjusted weights. The option can only be specified when weights or svyset are used to specify a weight variable.

madfactor(#) specifies the Fisher consistency factor to be applied to the median absolute deviation, if this statistic is selected for the calculation of the z-score. The default is 1.4826.

sfactor(#) specifies the Fisher consistency factor to be applied to the S-statistic, if this statistic is selected for the calculation of the z-score. The default is 1.1926.

qfactor(#) specifies the Fisher consistency factor to be applied to the Q-statistic, if this statistic is selected for the calculation of the z-score. The default is 2.2219.

small applies small samples correction for MAD, S and Q statistics.

Reporting

sformat(%fmt) specifies a format for the summary statistics panel.

pline(# | varname) specifies a poverty line, either as a scalar or as a variable. If pline() is specified, the output reports three poverty indices from the Foster, Greer and Thorbecke (1984) class,
namely the poverty headcount (H), poverty gap (PG), and poverty gap squared (PG2).

excel(filename [, replace]) exports the output table produced by the program in Excel workbook filename. If the replace option is specified, the existing filename is overwritten.

Graphics

graph(gtype [, twoway_options]) produces diagnostic plots. Any options documented in twoway_options other than by() can be specified. gtype can be chosen among the following:

qqplot plots the quantiles of the normalized variable against the quantiles of a Normal distribution with the same mean and variance (Quantile-Quantile plot). It uses Stata's qnorm. It can be used to assess the success of the normalization of varname.

itc([# :] itc_options) produces the Incremental Trimming Curve (ITC) for a statistic of interest (Mancini and Vecchi, 2021).
The ITC reports the value of the statistic of choice, as a function of how many extreme values are discarded (trimmed) from the distribution of varname.
Weights are adjusted at each iteration.
The ITC can be used to assess the sensitivity of the statistic of interest to the choice of the outlier detection threshold. By default,
the horizontal axis reports the number of trimmed observations as a percentage of all non-missing values of varname, but the number can be reported in absolute terms, too.

ifc([# :] ifc_options) produces the Influence Function Curve (IFC) for inequality indexes (Cowell and Flachaire, 2007). The IFC reports a measure of sensitivity of the index of interest to influential observations. The measure is a function of the i-th smallest or largest observation, i.e. IF(i) = [I-I(i)]/I, where I is the index estimate obtained using the full sample, and I(i) is the index estimate obtained using a sample from which the i-th observation has been omitted.

itc_options

# specifies the maximum percentage of trimmed observations. Default is 10%

absolute treats # as the number of trimmed observations

gini Gini index (default)

mean Sample mean

mld Mean Logarithmic Deviation

theil Theil index

cv2 Squared coefficient of variation

atk0 Atkinson index, A(0.125)

atk1 Atkinson index, A(1)

atk2 Atkinson index, A(2)

h Poverty headcount ratio

pg Poverty gap index

pg2 Poverty gap squared index

pline(varname | #) specifies the poverty line. It must be specified when h, pg, or pg2 are specified. If only pline(varname | #) is specified, then h is forced

ifc_options

# specifies the maximum # of large and small observations which, one at a time, are omitted for the calculation of the influence curve. Default is 10

gini Gini index (default)

mld Mean Logarithmic Deviation

theil Theil index

cv2 Squared coefficient of variation

atk0 Atkinson index, A(0.125)

atk1 Atkinson index, A(1)

atk2 Atkinson index, A(2)

Examples

Set up and default use

Load demo data

use hbs.dta, clear

Run outdetect using weights

outdetect pce [pweight=weight]

Specify survey settings before running outdetect

svyset [pweight=weight]

outdetect pce, replace

Flag outliers using different normalizations, z-scores, outlier detection thresholds

Flag outliers of pce using Box-Cox transformation and z-score based on the mean and variance

outdetect pce, norm(bcox) zscore(mean std) replace

Flag outliers of pce using the best normalizing transformation

outdetect pce, best replace

Only flag top outliers ("too large" values), and set outlier detection threshold to 2 (more "severe" than the default)

outdetect pce, out(top) alpha(2) replace

Format and export summary statistics, produce diagnostic graphs

Format summary statistics

outdetect pce, sformat(%10.0fc) replace

Report poverty estimates and export output table in Excel

outdetect pce, pline(300000) excel(demo, replace) replace

Generate Quantile-Quantile plot

outdetect pce, graph(qqplot) replace

Generate Incremental Trimming Curve (ITC) for the Gini index and poverty headcount ratio, discarding the 10% smallest and largest observations

outdetect pce, graph(itc)

outdetect pce, graph(itc(pline(300000)))

Generate ITC for the Gini index, discarding the 15 smallest and largest observations

outdetect pce, graph(itc(15:abs))

Generate ITC for the poverty gap index, discarding the 5% smallest and largest observations and export output table in Excel

outdetect pce, graph(itc(5:pg pline(300000))) excel(demo, replace)

Generate the Influence Function Curve (IFC) for the mean logarithm deviation index, discarding the 5 smallest and largest observations

outdetect pce, graph(ifc(5: mld))

Stored results

outdetect stores the following in r():

Scalars

r(N_raw) number of used observations

r(N_trimmed) number of observations after outliers trimming

r(bestnormalize) 1 if bestnormalize, 0 otherwise

r(pearson_df) ratio of the Pearson P statistic and df

r(alpha) threshold used for defining the outlier detection region

Macros

r(normalization) applied normalization

r(cmd) command name

Matrices

r(out) matrix containing information on the outliers' incidence

r(b) matrix containing statistics for the raw and trimmed variable

Reference

Belotti, F., G. Mancini, and G. Vecchi. 2022. Outlier Detection for Welfare Analysis. Unpublished manuscript, University of Rome “Tor Vergata”.

Mancini, G. and G. Vecchi. 2022.
On the Construction of the Consumption Aggregate for Inequality and Poverty Analysis.
Washington, DC: World Bank.

Box, G. E., and Cox, D. R. 1964. An analysis of transformations. Journal of the Royal Statistical Society: Series B (Methodological), 26(2), 211-243.

Foster, J., Greer, J., and Thorbecke, E. 1984. A class of decomposable poverty measures. Econometrica, 761-766.

Friedline, T., Masa, R. D., and Chowa, G. A. 2015. Transforming wealth: Using the inverse hyperbolic sine (IHS) and splines to predict youth’s math achievement. Social science research, 49, 264-287.

Rousseeuw, P. J., and Croux, C. 1993. Alternatives to the median absolute deviation. Journal of the American Statistical association, 88(424), 1273-1283.

Snedecor, G. W., Cochran, W. G. 1989. Statistical methods. Ames: Iowa State Univ. Press.

Yeo, I. and Johnson, R.A. 2000. A new family of power transformations to improve normality or symmetry. Biometrika, 87, 954-959.

Contact

To report any issues, please contact Giovanni Vecchi (giovanni.vecchi@uniroma2.it).

Authors

Federico Belotti

Department of Economics and Finance

University of Rome Tor Vergata

Rome, Italy

federico.belotti@uniroma2.it

Giulia Mancini

Department of Economics and Finance

University of Rome Tor Vergata

Rome, Italy

giulia.mancini@uniroma2.it

Giovanni Vecchi

Department of Economics and Finance

University of Rome Tor Vergata

Rome, Italy

giovanni.vecchi@uniroma2.it