help for obsofint -------------------------------------------------------------------------------
obsofint -- Displays observations of interest.
obsofint [varlist] [if] [in] [weight] [, zcutoff(#) whiskerlength(#) z tukey asymtukey idlist(varlist) sortid nosort loud summarize[(stats[, verbose])] generate(stub [, replace all]) list_options ]
fweights, aweights, and pweights are allowed.
obsofint is intended to help scan a large number of variables for unusual observations. These unusual observations are unusual in the sense that they either have a much smaller or a much larger value on a given variable than the bulk of the data. Such unusual observations are commonly referred to as outliers. While such observations are commonly referred to as outliers, we feel that this term is increasingly associated with the bad practice of automatically considering such observations as a problematic cases which should automatically be deleted. So instead we will use the term observations of interest. On the one hand, this terminology allows for the possibility that these observations are date entry errors, while on the other hand it also allows for the possibility that these observations are legitimate observations. In the former case deletion of that observation is only a measure of last resort, while in the latter case deletion would remove especially informative observations.
The observations of interest are identified using a criterium that is an adaptation of the commonly used Tukey bounds. In our experience these Tukey bounds flagged too many values as extreme values if a variable is either skewed or has a spike (a value that is very common). This is not surprising as these bounds were intended for normal/Gaussian-like variables, but it makes these bounds less useful for scanning a large set of variables for unusual observations. So, instead obsofint will use the following generalization of the Tukey bounds:
The traditional Tukey bounds are:
lb = Q_1 - 3*(Q_3 - Q_1), ub = Q_3 + 3*(Q_3 - Q_1)
The adjusted Tukey bounds are:
lb = Q_1 - 6*(Q_2 - Q_1), ub = Q_3 + 6*(Q_3 - Q_2)
lb and ub are respectively the lower and the upper bounds. Q_1, Q_2, and Q_3 are the first, second, and third quartile.
These adjusted Tukey bounds tend to lead to less false positives --- that is, less observations that are flagged as of interest that are actually perferctly normal --- when the data is skewed. However, these bounds do still lead to too many false positives when the variable contains a spike that is large enough to make either the first and second quartile, or the second and third quartile, or all three quartiles the same. If this happens, obsofint will automatically change the criterium of an observation of interest to a deviation of more than 3 standard deviation from the mean.
whiskerlength(#) specifies the number of inter-quartile ranges one needs to deviate from the 1st or 3rd quartile in order to be classified as an observation of interest, when the traditional Tukey bounds are used. The default is 3. In the adjusted Tukey ranges, that are default in obsofint, it specifies 1/2 of the number of distances between the lower quartile and the median or the higher quartile and the median. Notice that this way the traditional Tukey bounds and the adjusted tukey bounds lead to exactly the same results when the distribution of that variable is symetric.
zcutoff(#) specifies the number of standard deviations that an observation needs to deviate from the mean in order to be classified as an observation of interest. The default is 3.
z specifies that only the z-score criterium --- that is, the number of standard deviations that an observation deviates from the mean --- is to be used when identifying observations of interest. When this option is specified, one can not specify the whiskerlength() option.
tukey specifies that only the traditional Tukey bounds are to be used when identifying observations of interest. When this option is specified, one can not specify the zcutoff() option.
asymtukey specifies that only the adjusted Tukey bounds are to be used when identifying observations of interest. When this option is specified, one can not specify the zcutoff() option.
idlist(varlist) specifies the variables that are to be listed next to the extreme values. Typically these will be either identification numbers and/or variables that might explain why an observation might be exceptional.
sortid specifies that observations are sorted by the variables specified in idlist(). The idlist() option must thus be specified when specifying the sortid options. The default is that the observations are sorted by the variable being listed.
nosort specifies that the observation are not sorted. The default is that the observations are sorted by the variable being listed.
If the nosort option is specified the observation numbers will be displayed as is normal in list. When the nosort option is specified, the observation number will be displayed as an variable with the name obs_nr unless there is already a variable with that name. In that case it will use the name obs unless there is also a variable with that name. In that case the name _n will be used.
loud specifies that obsofint will display a message for each variable that does not have any observations flagged as of interest. By default, obsofint displays nothing for those variables.
summarize[(stats[, verbose])] specifies that for those variables that contain observations of interest, the statistics stats will be displayed in the report. If no statistics have been it will display N, mean, sd, min, p25, p50, p75, and max. One or more of the following statistics may be specified: N, sum_w, mean, Var, sd, skewness, kurtosis, sum, min, max, p1, p5, p10, p25, p50, p75, p90, p95, p99. The verbose suboption
generate(stub [, replace all]) specifies that indicator variables are be created for each variable containg variables of interest, which will be 1 if that observation is of interest and 0 otherwise. These indicator variables will be called stub_variablename. If these variables already exist obsofint will exit with an error unless the replace sub-option is specified, in which case the existing indicator variables will be overwritten. If the all suboption is specified these indicator variables will be created for all variables checked by obsofint, regardless whether observations of interest were found or not.
list_options all options for list can also be specified for obsofint and will be used when listing observations of interest.
r(result) Contains a matrix with a row for each variable in varlist. The first column shows the number of observation classified as of interest, the second and third columns show the lower and upper bound used to classify the observations. These bounds will be missing when the variable is a constant. The last three collumns indicate whether the z-score, adjusted Tukey bounds, or traditional Tukey bounds criterium was used to identify observations of interest. The criterium used is represented by a 1 and the remaining criteria will receive a 0. If a variable is a constant, all criteria will receive a 0.
sysuse auto, clear obsofint, idlist(make)
(click to run)
sysuse auto, clear obsofint price - foreign, loud idlist(make) sum
(click to run)
sysuse auto, clear obsofint price - foreign, idlist(make) tukey
(click to run) Authors
Maarten L. Buis Universitaet Tuebingen Institut fuer Soziologie email@example.com
Ronnie Babigumira Center for International Forestry Network (CIFOR) The Poverty Environment Network (PEN) firstname.lastname@example.org
obsofint was written while Maarten Buis was visiting CIFOR as a consultant to work on the PEN project.
Online: lv iqr (if installed)