*! version 0.2.1 30Jan2020

Title

gstats transform -- Apply statistical functions by group using C for speed

Important: Please run gtools, upgrade to update gtools to
the latest stable version.

Syntax

gstats transform
clist
[if] [in]
[weight]
[,
options]

where clist is either

[(stat)]
varlist
[ [(stat)] ... ]

[(stat)] target_var=varname
[target_var=varname ...]
[ [(stat)] ...]

or any combination of the varlist or target_var forms, and
stat is one of

demean subtract the mean (default)

demedian subtract the median

normalize (x - mean) / sd

standardize same as normalize

moving stat [# #] moving statistic stat; # specify the relative bounds (see below)

range stat [...] range statistic stat for observations within specified interval (see below)

cumsum [+/- [varname]] cumulative sum, optionally ascending (+) or descending (-) (optionally +/- by varname)

shift [[+/-]#] lags (-#) and leads (+#); unsigned numbers are positive (i.e. leads)

rank rank observations; use option ties() to specify how ties are handled

Some of the above transformations allow specifying various
options as part of their name. This is done to allow the user to request
various versions of the same transformation. However, this is not
required. The user can specify a global option that will be used for
all the corresponding transformations:

moving stat window()

range stat interval()

cumsum cumby()

shift shiftby()

Note gstats moving and gstats range are aliases
for gstats transform. In this case all the requested statistics
are assumed to be moving or range statistics, respectively. Finally,
moving and range may be combined with any one of the
folloing:

mean means (default)

geomean geometric mean (missing if var has any negative values)

count number of nonmissing observations

nmissing number of missing observations

sum sums

rawsum sums, ignoring optionally specified weights (note: zero-weighted obs are still excluded)

nansum sum; returns . instead of 0 if all entries are missing

rawnansum rawsum; returns . instead of 0 if all entries are missing

median medians (same as p50)

p#.# arbitrary quantiles

p1 1st percentile

p2 2nd percentile

... 3rd-49th percentiles

p50 50th percentile (same as median)

... 51st-97th percentiles

p98 98th percentile

p99 99th percentile

iqr interquartile range

sd standard deviation

variance variance

cv coefficient of variation (sd/mean)

select# #th smallest

select-# #th largest

rawselect# #th smallest, ignoring weights

rawselect-# #th largest, ignoring weights

max maximums

min minimums

range range = max - min

first first value

last last value

firstnm first nonmissing value

lastnm last nonmissing value

semean standard error of the mean (sd/sqrt(n))

sebinomial standard error of the mean, binomial (sqrt(p(1-p)/n)) (missing if source not 0, 1)

sepoisson standard error of the mean, Poisson (sqrt(mean / n)) (result rounded to nearest integer)

skewness Skewness

kurtosis Kurtosis

gini Gini coefficient (negative truncated to 0)

gini dropneg Gini coefficient (negative values dropped)

gini keepneg Gini coefficient (negative values kept; the user is responsible for the interpretation of the Gini in this case)

Interval format

range stat must specify an interval or use the interval(...)
option. The interval must be of the form

#[statlow] #[stathigh] [var]

This computes, for each observation i, the summary statistic stat
among all observations j of the source variable such that

var[i] + # * statlow(var) <= var[j] <= var[i] + # * stathigh(var)

if var is not specified, it is taken to be the source variable itself.
statlow and stathigh are summary statistics computed based on
every value of var. If they are not specified, then # is used by
itself to construct the bounds, but # may be missing (.) to mean no
upper or lower bound. For example, given some variable x with N observations,
we have

Input -> Meaning

-2 2 time -> j: time[i] - 2 <= time[j] <= time[i] + 2
i.e. stat within a 2-period time window

-sd sd -> j: x[i] - sd(x) <= x[j] <= x[i] + sd(x)
i.e. stat for obs within a standard dev

Moving window format

moving stat must specify a relative range or use the window(# #)
option. The relative range uses a window defined by the observations. This
would be equivalent to computing time series rolling window statistics
using the time variable set to _n. For example, given some variable
x with N observations, we have

Input -> Range

-3 3 -> x[i - 3] to x[i + 3]
-3 . -> x[i - 3] to x[N]
. 3 -> x[1] to x[i + 3]
-3 -1 -> x[i - 3] to x[i - 1]
-3 0 -> x[i - 3] to x[i]
5 10 -> x[i + 5] to x[i + 10]

and so on. If the observation is outside of the admisible range
(e.g. -10 10 but i = 5) the output is set to missing. If you
don't specify a range in (moving stat) then the range in window(# #)
is used.

Options

Common Options
by(varlist) Group statistics by variable.

replace Allow replacing existing variables.

wildparse Allow rename-style syntax in target naming.

labelformat Custom label engine: (#stat#) #sourcelabel# is the default.

labelprogram(str) Program to parse labelformat (see examples).

autorename[()] Automatically name targets based on requested stats. Default is #source#_#stat#.

nogreedy Use slower but memory-efficient (non-greedy) algorithm.

types(str) Override variable types for targets (use with caution).

Command Options
window(lower upper) With moving stat. Relative observation range for moving statistics (if not specified in call). E.g. window(-3 1) means from 3 lag to 1 lead. window(. #) and window(# .) mean from the start and through the end.

interval(#[stat] #[stat] [var]) With range stat. Interval for range statistics that don't specify their own interval.

cumby([+/- [varname]]) With cumsum. Sort options for cumsum variables that don't specify their own.

shiftby([+/-]#) With shift. Lag or lead when to use shift is requested without specifying a number.

ties(str) With rank. How to break ties for rank. default assigns the average rank; unique breaks ties arbitrarily; stableunique breaks ties using the order values appear in the data; field counts the number of values greater than; track counts the number of values less than.

Gtools
compress Try to compress strL to str#.

forcestrl Skip binary variable check and force gtools to read strL variables.

verbose Print info during function execution.

bench[()] Benchmark various steps of the plugin. Optionally specify depth level.

hashmethod(str) Hash method (default, biject, or spooky). Intended for debugging.

oncollision(str) Collision handling (fallback or error). Intended for debugging.

aweights, fweights, iweights, and pweights
are allowed.

Description

gstats transform applies various statistical transformations
to input data. It is similar to gcollapse, merge or gegen but
for individual-level transformations. That is, gcollapse takes an
input variable and procudes a single statistic; gstats transform
applies a function to each element of the input variable. For example,
subtracting the mean.

Every function available to gstats transform can be called via
gegen. Further, note that while not every function will use weights
in their computations (e.g. shift ignores weights in the actual
transformation), if weights are specified they will be used to flag
acceptable observations (i.e. missing, zero, and, except for iweights,
negative observations get excluded).

Examples

See the
online documentation
for examples.

Author

Mauricio Caceres

mauricio.caceres.bravo@gmail.com

mcaceresb.github.io

Website

gstats is maintained as part of the [R] gtools project at github.com/mcaceresb/stata-gtools

Acknowledgment

gtools was largely inspired by Sergio Correia's ftools:
https://github.com/sergiocorreia/ftools.

The OSX version of gtools was implemented with invaluable help from @fbelotti;
see https://github.com/mcaceresb/stata-gtools/issues/11.

Also see

help for
gegen;
gcollapse;
gtools