*! version 2.12 12-Jun-20233, Dirk Enzmann
help nb_adjust

Title

nb_adjust -- adjust or remove outliers of a variable assumed
to have a negative binomial distribution

Syntax

nb_adjust varname [if] [in] [, options ]

options

generate(newvar) generate newvar with outliers
adjusted or removed

small(#) smallest value to define outliers (default: 0)

large(#) value assumed to be large and no outlier (default: 0)

threshold(#) fix threshold to define outliers (default: not
fixed)

limit(#) values beyond limit are extremes and will be
removed (default: none)

seed(#) initial value of random-number seed (default: not
set)

replicates(#) number of replicates of random numbers (default: 250)

censor censor outliers instead of adjustment (default: adjust)

remove remove outliers instead of adjustment (default: adjust)

nodetail suppress details (default: show details)

replace replace contents of newvar if newvar
exists already

by is allowed (see by)

Description

nb_adjust identifies and adjusts (or removes) outliers
of varname assuming that the values of varname have a
negative binomial distribution. Per default a value is defined as an outlier
if its expected frequency is less than 0.5 (rule-based outlier definition).

nb_ajdust calculates the threshold to define an outlier by
estimating the mean and overdispersion parameter of varname and by
using the parameters mu = exp(_b[_cons]) and size = mu/e(delta)
obtained by -nbreg varname, dispersion(constant)- as follows:

threshold = max(counts) of counts such that

round(n * nbinomialp(size,counts,prob)) > 0

with counts = 0..max(varname), n = sample size,
and prob = size/(size + mu).

Alternatively, the user can fix the threshold defining an outlier by
using the option threshold(#).

Per default outliers are values greater 0 which are greater than the
rule-based or user-fixed threshold. When using the rule-based definition of
outliers it is possible that values will be defined as outliers which the user
nevertheless wants to treat as "normal". By using the option small(#)
(default: 0) the user can restrict the value to define outliers to a
minimum: When using this option, only values greater than the maximum of
(small, threshold) are defined as outliers.

nb_adjust adjusts outliers by replacing its values by random draws
from a negative binomial distribution with parameters mu and size as
estimated using the original values of varname. Replacement values
are ordered by size such as to preserve the rank order of outlying cases. To
make sure that the randomly drawn values have a minimal size, the user can apply
the option large(#) (default: 0): Using this option, the lower bound of
replacement values is the minimum of (large, threshold) (i.e. in
cases of threshold < large the minimum of replacement values is
threshold instead of large). Note, however, that the absolute
minimum of replacement values is always small. The upper bound of
replacement values is the original value.

The adjustment of outliers and the rule-based definition of the threshold
defining outliers are based on the observed values of varname. Using
the option limit(#), extreme values greater than limit can be
eliminated from the observed values before estimating the mean and overdispersion
parameter of varname. Excluding extremes by using the option
limit(#), only the remaining values are used to define the rule-based
outlier threshold and to define the negative binomial distribution from which to
draw replacement values of outliers. If the option generate(newvar) is
used, values of varname > limit will be set to the extended
missing value .r in variable newvar.

To allow a replication of the random draws, the user can set the initial
value of the random-number seed to any number between between 0
and 2^31-1 (2,147,483,647) by using the option seed(#).

Per default the random draws for the adjustment of outliers will be
replicated 250 times and averages over the replications rounded to the next
integer will be used as replacement values. The option replicates(#) allows
to specify the number of replicates (minimum 0 = no replicates).

Instead of adjusting outlying values, outliers can be censored (i.e. set
to the outlier threshold) using the option censor or removed (i.e. set to
the extended missing value .o in variable newvar) by using the
option remove. In this case, the option large(#) will have no
effect. Note, however, that values to be censored or removed must always
be > small.

An example showing how to apply outlier detection and outlier adjustment of
negative binomial distributed counts for the estimation of rates of reporting
criminal victimizations to the police is shown in Section 3 of Enzmann (2023).

Options

Main

generate(newvar) specifies the name newvar of the variable
with outliers of varname adjusted or removed.

small(#) restricts the value to define outliers to a minimum
(default: 0). If the rule-based outlier threshold is less than small, only
values > small will be treated as outliers. Note that the user-fixed
outlier threshold must not be less than small.

large(#) makes sure that the randomly drawn values to replace outliers
have a minimal size. Using this option the lower bound of replacement values is
the minimum of (large, threshold). In cases of threshold <
large the minimal size of replacement values is threshold instead of
large.

threshold(#) sets the threshold of outliers to a user-fixed value (default:
rule-based threshold). Using this option overrides the default of nb_adjust
which determines the threshold of outliers such that the expected frequency of an
outlying value is less than 0.5. Note that the user-fixed value of threshold
must not be smaller than the value specified with the option small(#).

limit(#) serves to exclude extreme values > limit when determining
the mean and overdispersion parameter of varname. Values > limit
will temporarily be set to missing and will be set to the extended missing
value .r when generating newvar. This option can be used to eliminate
the influence of extreme values on the rule-based definition of outliers and
their randomly drawn replacement values.

seed(#) specifies the initial value of the random-number seed (default: not
set). To enable a replication of the random numbers drawn when adjusting outliers,
seed can be set to any number between between 0 and 2^31-1 (2,147,483,647).

replicates(#) specifies the number of replicates for the adjustment of
outliers (default: 250). Replacement values are the averages over the replications
rounded to the next integer. The minium value of replicates(#) is
0 (no replicates).

censor sets all outliers to the constant value of the outlier threshold instead
of adjusting outlying values by random draws from a negative binomial distribution. Note
that only values greater than the maximum of (threshold, small) (see option
small(#)) will be censored.

remove sets outliers to the extended missing value .o instead of
adjusting outlying values by random draws from a negative binomial distribution. Note
that only values greater than the maximum of (threshold, small) (see option
small(#)) will be removed by setting them to missing.

nodetail allows to reduce details of the output.

replace replaces the variable specified by generate(newvar) if
newvar exists already.

Example

Using an open response question, 12 year old students were asked to
indicate the number of times their family had moved to a different place. The
frequency distribution of the count variable "moves" shows at least one case
with a rather implausible value of 32.

To replicate the example, copy and paste the 26 lines between -clear- and
-tab moves, missing- into Stata's command window:

clear
input moves freq
0 486
1 763
2 315
3 281
4 163
5 88
6 40
7 27
8 9
9 5
10 2
11 2
12 1
13 2
15 2
17 1
18 2
24 1
25 1
32 1
. 12
end
expand freq
tab moves, missing

nb_adjust is used to create two new variabels ("moves_adj" and
"moves_rem") with outlying values adjusted and removed, resp. Before using the
default rule of nb_adjust to determine the threshold of outliers and to
randomly draw replacement values, extremely implausible values > 30 will be
removed. A number of three moves are assumed to be "normal", thus the option
small(3) will be used to restrict outliers to values > 3. Fifteen moves
are assumed to be "large", therefore replacement values of outliers are
constrained to be at least 15.

Thus, three options specifiying small values, large values, and the limit
to extreme values will be used for adjusting outliers (large values need not be
specified when removing outliers). The option seed(#) will be set to the
arbitrary value of 4 to enable a replication of the random draws of replacement
values:

nb_adjust moves, g(moves_adj) sm(3) la(15) li(30) seed(4)
nb_adjust moves, g(moves_rem) sm(3) li(30) rem

The default rule of nb_adjust determined 12 to be the rule-based
threshold of outliers (overall, 9 values or 0.4% of sample are > 12 and thus
defined as outliers). Because 12 is greater than the minimal size of values
defined as "normal" by using option small(3), all 9 cases will have their
values adjusted by random draws from a negative binomial distribution or will be
removed by setting them to the extended missing value .o.

Because the outlier threshold of 12 is smaller than the minimal size of
adjusted values specified by the option large(15), the lower bound
of adjusted values was reduced to the outlier threshold of 12. The commands
fre (if necessary, fre
will be installed by the first call to nb_adjust) and sum show
the effect of adjusting and removing the 9 outliers:

fre moves*
sum moves*

Saved Results

nb_adjust saves the following in r():

Scalars

r(small) minimum value to define outliers

r(large) specified minimal size of replacement values

r(limit) limit to define extreme values (0 = not used)

r(seed) initial value of the random number seed (-1 = not used)

r(repl) number of replicates of the random draws

r(N) number of valid cases (cases not greater than limit) of
varname

r(mu) mean of varname

r(size) mu/delta of varname as determined by
-nbreg varname, dispersion(constant)-

r(threshold) threshold defining values as outliers

r(nout) number of outliers of varname

r(percout) percentage of outliers of varname

r(low) lower bound used for adjustment values

r(nadj) number of outliers adjusted (or removed)

Macros

r(varname) name of variable used

r(newvar) name of new variable with values adjusted or removed

r(adj) handling of outliers (adjusted, censored, or removed)

r(method) method to define outliers (rule-based or fixed)

References

Enzmann, D. (2023). Reporting Rates as an Indicator of Ignorance: Issues of Measurement and Design. Hamburg: University of Hamburg, Institute of
Criminal Sciences. [http://dx.doi.org/10.13140/RG.2.2.20133.68328]

Requires

nb_adjust requires the SSC packages fre,
elabel,
and moremata. If
necessary they will be installed by the first call of nb_adjust.

Author

Dirk Enzmann

Institute of Criminal Sciences, Hamburg

email: mailto:dirk.enzmann@uni-hamburg.de