*! version 0.2.1 30Jan2020

Title

gstats summarize -- Summary statistics by group using C for speed

Important: Please run gtools, upgrade to update gtools to
the latest stable version.

Syntax

gstats summarize
varlist
[if] [in]
[weight]
[, by(varlist) options]

gstats tabstat
varlist
[if] [in]
[weight]
[, by(varlist) options]

gstats tabstat and gstats summarize are fast, by-able
alternatives to tabstat and summarize, detail.
If gstats summarize is called with by() or tab, a table
in the style of tabstat is produced that inclues all the summary
statistics included by default in summarize, detail.

Note the prefixes by, rolling, statsby are
not supported. To compute a table of statistics by a group
use the option by(). With by(), gstats tab is also
faster than gcollapse.

Tabstat Options
by(varlist) Group statistics by variable.

statistics:(stat [...]) Report
specified statistics; default for tabstat is count, sum, mean, sd, min, max.

columns(stat|var) Columns are statistics (default) or variables.

prettystats Pretty statistic header names

labelwidth(int) Max by variable label/value width.

format[(%fmt)]
Use format to display summary stats; default %9.0g

Summarize Options
nodetail Do not display the full set of statistics.

meanonly Calculate only the count, sum, mean, min, max.

by(varlist) Group by variable; all stats are computed but output is in the style of tabstat.

separator(#) Draw separator line after every # variables; default is separator(5).

tabstat Compute and display statistics in the style of tabstat.

Common Options
matasave[(str)] Save results in mata object (default name is GstatsOutput)

pooled Pool varlist

noprint Do not print

format Use variable's display format.

nomissing With by(), ignore groups with missing entries.

Gtools Options
compress Try to compress strL to str#.

forcestrl Skip binary variable check and force gtools to read strL variables.

verbose Print info during function execution.

bench[(int)] Benchmark various steps of the plugin. Optionally specify depth level.

hashmethod(str) Hash method (default, biject, or spooky). Intended for debugging.

oncollision(str) Collision handling (fallback or error). Intended for debugging.

aweights, fweights, iweights, and pweights are
allowed (see [U] 11.1.6 weight for more on the way Stata
uses weights).

Description

gstats tab and gstats sum are mainly designed to report
statistics by group. It does not modify the data in memory,
so it is a nice alternative to gcollapse when there are few
groups and you want to compute summary stats more quickly.

gstats sum by default computes the staistics that are reported by
sum, detail and without by() it is anywhere from 5 to 40
times faster. The lower end of the speed gains are for Stata/MP, but
sum, detail is very slow in versions of Stata that are not multi-threaded.
The behavior of plain summarize and summarize, meanonly
can be recovered via options nodetail and meanonly, but Stata
is not specially slow in this case. Hence they are mainly included for
use with by(), where gstats sum is again faster.

gstats tab should be faster than tabstat even without
groups, but the speed gains are largest with even a modest number of
levels in by(). Furthermore, an arbitrary number of grouping
variables are allowed. Note that with a very large numer of groups,
tabstat's runtime seems to scale non-linearly, while gstats tab
will execute in a reasonable time.

gstata tab does not store results in r(). Rather, the option matasave
is provided to store the full set of summary statistics and the by variable
levels in a mata class object called statsOutput (the name of the object
can be changed via matasave(name)). Run mata GstatsOutput.desc()
after gstats tab, matasave for details. The following helper functions are provided:

string scalar getf(j, l, maxlbl)
get formatted (j, l) entry from by variables up to maxlbl characters

real matrix getnum(j, l)
get (j, l) numeric entry from by variables

string matrix getchar(j, l,| raw)
get (j, l) numeric entry from by variables; raw controls whether to null-pad entries

real rowvector getOutputRow(j)
get jth output row

real colvector getOutputCol(j)
get jth output column by position

real matrix getOutputVar(var)
get jth output var by name

real matrix getOutputGroup(j)
get jth output group

The following data is stored GstatsOutput:

summary statistics
------------------

real matrix output
matrix with output statistics; J x kstats x kvars

real scalar colvar
1: columns are variables, rows are statistics; 0: the converse

real scalar ksources
number of variable sources (0 if pool is true)

real scalar kstats
number of statistics

real matrix tabstat
1: used tabstat; 0: used summarize

string rowvector statvars
variables summarized

string rowvector statnames
statistics computed

real rowvector scodes
internal code for summary statistics

real scalar pool
pooled source variables

variable levels (empty if without -by()-)
-----------------------------------------

real scalar anyvars
1: any by variables; 0: no by variables

real scalar anynum
1: any numeric by variables; 0: all string by variables

real scalar anychar
1: any string by variables; 0: all numeric by variables

string rowvector byvars
by variable names

real scalar kby
number of by variables

real scalar rowbytes
number of bytes in one row of the internal by variable matrix

real scalar J
number of levels

real matrix numx
numeric by variables

string matrix charx
string by variables

real scalar knum
number of numeric by variables

real scalar kchar
number of string by variables

real rowvector lens
> 0: length of string by variables; <= 0: internal code for numeric variables

real rowvector map
map from index to numx and charx

printing options
----------------

void printOutput()
print summary table

real scalar maxlbl
max by variable label/value width

real scalar pretty
print pretty statistic names

real scalar usevfmt
use variable format for printing

string scalar dfmt
fallback printing format

real scalar maxl
maximum column length

void readDefaults()
reset printing defaults

Statistics

statistics(statname [...])
specifies the statistics to be displayed; the default with tabstat
is equivalent to specifying statistics(mean). (stats()
is a synonym for statistics().) Multiple statistics
may be specified and are separated by white space, such as
statistics(mean sd). Available statistics are

statname Definition

mean mean

geomean geometric mean (missing if var has any negative values)

count count of nonmissing observations

n same as count

nmissing number of missing observations

percent percentage of nonmissing observations

nunique number of unique elements

sum sum

rawsum sum, ignoring optionally specified weights (note: zero-weighted obs are still excluded)

nansum sum; returns . instead of 0 if all entries are missing

rawnansum rawsum; returns . instead of 0 if all entries are missing

median median (same as p50)

p#.# arbitrary quantiles

p1 1st percentile

p2 2nd percentile

... 3rd-49th percentiles

p50 50th percentile (same as median)

... 51st-97th percentiles

p98 98th percentile

p99 99th percentile

iqr interquartile range = p75 - p25

q equivalent to specifying p25 p50 p75

sd standard deviation

variance variance

cv coefficient of variation (sd/mean)

select# #th smallest

select-# #th largest

min minimum (same as select1)

max maximum (same as select-1)

range range = max - min

first first value

last last value

firstnm first nonmissing value

lastnm last nonmissing value

semean standard error of mean (sd/sqrt(n))

sebinomial standard error of the mean, binomial (sqrt(p(1-p)/n))

sepoisson standard error of the mean, Poisson (sqrt(mean))

skewness skewness

kurtosis kurtosis

gini Gini coefficient (negative truncated to 0)

gini|dropneg Gini coefficient (negative values dropped)

gini|keepneg Gini coefficient (negative values kept; the user is responsible for the interpretation of the Gini in this case)

Examples

See the
online documentation
for examples.

Author

Mauricio Caceres

mauricio.caceres.bravo@gmail.com

mcaceresb.github.io

Website

gstats is maintained as part of the [R] gtools project at github.com/mcaceresb/stata-gtools

Acknowledgment

gtools was largely inspired by Sergio Correia's ftools:
https://github.com/sergiocorreia/ftools.

The OSX version of gtools was implemented with invaluable help from @fbelotti;
see https://github.com/mcaceresb/stata-gtools/issues/11.

Also see

help for
summarize;
tabstat;
gtools