*! version 1.2.0 20Mar2019

Title

gtoplevelsof -- Quickly tabulate most common levels of variable list.

Syntax

gtoplevelsof
varlist
[if] [in]
[weight]
[, options]

Summary Options
ntop(int) Display ntop most common levels (negative shows least common; . shows every level).

freqabove(int) Only count freqs above this level.

pctabove(real) Only count freqs that represent at least % of the total.

matasave[(str)] Save results in mata object (default name is GtoolsByLevels)

Toggles
missrow Add row with count of missing values.

groupmissing Count rows with any variable missing as missing.

nomissing Case-wise exclude rows with missing values from frequency count.

noother Do not group rest of levels into "other" row.

nongroups Do not specify number of groups in "other" row.

alpha Sort the top levels of varlist by variables instead of frequencies.

silent Do not display the top levels of varlist.

Display Options
pctfmt(format) Format for percentages.

otherlabel(str) Specify label for row with "other" count.

missrowlabel(str) Specify the label for the row with "missing" count.

varabbrev(int) Abbreviate variables (which are displayed as a header to their levels) .

colmax(numlist) Specify width limit for levels (can be single number of variable-specific).

colstrmax(numlist) Specify width limit for string variables (can be single number of variable-specific).

colseparate(separator) Column separator; default is double blank " ".

numfmt(format) Format for numeric variables. Default is %.8g (or %16.0g with matasave).

novaluelabels Do not replace numeric variables with their value labels.

hidecontlevels If a level is repeated in the subsequent row, display a blank.

levelsof Options
local(macname) insert top levels in the local macro macname

separate(separator) separator for the values of returned list; default is a space

Gtools
compress Try to compress strL to str#.

forcestrl Skip binary variable check and force gtools to read strL variables.

verbose Print info during function execution.

bench[()] Benchmark various steps of the plugin. Optionally specify depth level.

hashmethod(str) Hash method (default, biject, or spooky). Intended for debugging.

oncollision(str) Collision handling (fallback or error). Intended for debugging.

aweights, fweights, and pweights are allowed, in which
case the top levels by weight are printed (see [U] 11.1.6 weight)

Description

gtoplevelsof (alias gtop) displays a table with the
frequency counts, percentages, and cummulative counts and %s of the
most common levels of varlist that occur in the data. It is similar
to the user-written group with the select otpion or to
contract after keeping only the largest frequency counts.

Unlike contract, it does not modify the original data and instead prints the
resulting table to the console. It also stores a matrix with the frequency
counts and stores the levels in the macro r(levels).

gcontract is part of the [R] gtools project.

Options

Summary Options

ntop(int) Number of levels to display. This can be negative;
in that case, the smallest frequencies are displayed. Note cummulative
percentages and counts are computed within each generated table,
so for the smallest groups the table would display the cummulative
count for those frequencies, in descending order. . displays
every level from most to least frequent; -. displays every level
from least to most frequent.

freqabove(int) Skip frequencies below this level then determining the
largest levels. So if this is 10, only frequencies above 10 will be displayed
as part of the top frequencies. If every frequency that would be displayed is
above this level then this option has no effect.

pctabove(real) Skip frequencies that are a smaller percentage of the
data than pctabove. If this is 10, then only frequencies that represent
at least 10% of all observations are displayed as part of the top frequencies.
If every frequency that would be displayed is at least this percentage of the
data then this option has no effect.

matasave[(str)]Save results in mata object (default
name is GtoolsByLevels). See GtoolsByLevels.desc() for more.
This object contains the raw variable levels in numx and charx
(since mata does not allow matrices of mixed-type). The levels are saved
as a string in printed (with value labels correctly applied) unless
option silent is also specified. Last, the frequencies matrix is saved
in toplevels.

Toggles

missrow Add row with count of missing values. By default,
missing rows are treated as another group and will be displayed as part
of the top levels. With multiple variables, only rows with all values
missing are counted here unless groupmissing is also passed. If
this option is specified then a row is printed after the top levels
with the frequency count of missing rows.

groupmissing This option specifies that a missing row is a
row where any of the variables have a missing value. See missrow.

nomissing Case-wise exclude rows with missing values from
frequency count. By default missing values are treated as another level.

noother By default a row is printed after the top levels
with the frequency count from groups not in the top levels and not
counted as missing. This option toggles display of that row.

nongroups By default the number of groups comprising the
"Other" and "Missing" rows are printed as part of the "Other" and
"Missing" row labels (should they appear; for the missing row this
is only printed if more than 1 missing value type is present). This
option toggles display of the number of groups represented.

alpha Sort the top levels of varlist by variables instead
of frequencies. Note that the top levels are still extracted; this just
affects the final sort order. To sort in inverse order, just pass
gtop -var1 -var2 ....

silent Do not display the top levels of varlist. With
option matasave it also does not store the printed levels in a
separate string matrix.

Display Options

pctfmt(format) Print format for percentage columns.

otherlabel(str) Specify label for row with the count of the
rest of the levels.

missrowlabel(str) Specify the label for the row the count of
the "missing" levels.

varabbrev(int) Variables names are displayed above their
groups. This option specifies that variables should be abbreviated to at
most varabbrev characters. This is ignored if it is smaller than 5.

colmax(numlist) Specify width limit for levels (can be single
number of variable-specific).

colstrmax(numlist) Specify width limit for string variables (can
be single number of variable-specific). Ths overrides colmax for strings
and allows the user to specify string and number widths sepparately. (Also see
numfmt(format))

numfmt(format) Format for numeric variables. Default is %.8g
(or %16.0g with matasave). By default the number levels are formatted
in C, so this must be a valid format for the C internal printf. The syntax
is very similar to mata's printf. Some examples are: %.2f, %10.6g, %5.0f, and
so on. With option matasave these are formatted in mata, and the format can
be any mata number format.

colseparate(separator) Column separator; default is double blank " ".

novaluelabels Do not replace numeric variables with their value
labels. Value label widths are governed by colmax and NOT colstrmax.

hidecontlevels If a level is repeated in the subsequent row,
display a blank. This is only done if both observations are within the same
outer level.

levelsof Options

local(macname) inserts the list of levels in local macro
macname within the calling program's space. Hence, that macro will
be accessible after gtoplevelsof has finished. This is helpful for
subsequent use. Note this uses colseparate to sepparate columns. The
default is " " so be careful when parsing! Rows are enclosed in double quotes
(`""') so parsing is possible, just not trivial.

separate(separator) specifies a separator
to serve as punctuation for the values of the returned list.
The default is a space. A useful alternative is a comma.

Gtools

compress Try to compress strL to str#. The Stata Plugin Interface
has only limited support for strL variables. In Stata 13 and earlier
(version 2.0) there is no support, and in Stata 14 and later (version
3.0) there is read-only support. The user can try to compress strL
variables using this option.

forcestrl Skip binary variable check and force gtools to read strL
variables (14 and above only). Gtools gives incorrect results when there is binary data in strL variables.
This option was included because on some windows systems Stata detects
binary data even when there is none. Only use this option if you are
sure you do not have binary data in your strL variables.

verbose prints some useful debugging info to the console.

benchmark and benchmarklevel(int) print how long in
seconds various parts of the program take to execute. The user can also
pass bench(int) for finer control. bench(1) is the same
as benchmark but bench(2) and bench(3) additionally print
benchmarks for internal plugin steps.

hashmethod(str) Hash method to use. default automagically
chooses the algorithm. biject tries to biject the inputs into the
natural numbers. spooky hashes the data and then uses the hash.

oncollision(str) How to handle collisions. A collision should never
happen but just in case it does gtools will try to use native commands.
The user can specify it throw an error instead by passing oncollision(error).

Remarks

gtoplevelsof has the main function of displaying the most common levels
of varlist. While tab is great, it cannot handle a large number
of levels, and it prints ALL the levels in alphabetical order.

Very often when exploring data I just want to have a quick look at the largest
levels of a variable that may have thousands of levels in a data set with
millions of rows. gcontract and gcollapse are great but they
modify the original data and doing a lot of subsequent preserve, sort, restore
gets very slow very fast.

I have found this command extremely helpful when exploring big data.
Specially if a string is not clean, then having a look at the largest
values or the largest values that match a pattern is very helpful.

Examples

See the
online documentation
for more examples.

. sysuse auto

. gtoplevelsof rep78

. gtoplevelsof rep78, missrow local(toplevels)

. gtop rep78, colsep(", ")

. gtop foreign rep78, ntop(3) missrow

Stored results

gtoplevelsof stores the following in r():

Macros

r(levels) list of top (most common) levels (rows); not with matasave

r(matalevels) name of GtoolsByLevels mata object; only with matasave

Scalars

r(N) number of non-missing observations

r(J) number of groups

r(minJ) largest group size

r(maxJ) smallest group size

r(ntop) number of top levels

r(nrows) number of rows in toplevels

r(alpha) sorted by levels intead of frequencies

Matrices

r(toplevels) Table with frequency counts and percentages.

The missing and other rows are stored in the matrix with IDs 2 and 3,
respectively. With matasave, the following data is stored in GtoolsByLevels:

real scalar anyvars
1: any by variables; 0: no by variables

real scalar anychar
1: any string by variables; 0: all numeric by variables

real scalar anynum
1: any numeric by variables; 0: all string by variables

string rowvector byvars
by variable names

real scalar kby
number of by variables

real scalar rowbytes
number of bytes in one row of the internal by variable matrix

real scalar J
number of levels

real matrix numx
numeric by variables

string matrix charx
string by variables

real scalar knum
number of numeric by variables

real scalar kchar
number of string by variables

real rowvector lens
> 0: length of string by variables; <= 0: internal code for numeric variables

real rowvector map
map from index to numx and charx

real rowvector charpos
position of kth character variable

string matrix printed
formatted (printf-ed) variable levels (not with option -silent-)

real matrix toplevels
frequencies of top levels; missing and other rows stored with ID 2 and 3.

Author

Mauricio Caceres Bravo

mauricio.caceres.bravo@gmail.com

mcaceresb.github.io

Website

gtoplevelsof is maintained as part of gtools at github.com/mcaceresb/stata-gtools

Acknowledgment

This project was largely inspired by Sergio Correia's ftools:
https://github.com/sergiocorreia/ftools.

The OSX version of gtools was implemented with invaluable help from @fbelotti;
see https://github.com/mcaceresb/stata-gtools/issues/11.

Also see

help for
gcontract,
glevelsof,
gtools;
flevelsof (if installed),
ftools (if installed)