*! version 0.5.0 26Jan2020

Title

greshape -- Fast alternative to reshape using C for speed.

Important: Please run gtools, upgrade to update gtools to
the latest stable version.

Syntax

greshape is a fast alternative to reshape that additionally
implements the equivalents to R's spread and gather from
tidyr. Further, it allows an arbitrary number of grouping by variables
(i()) and keys (j()).

Normal syntax

In R parlance, j() is called keys(), which is
frankly much clearer to understand. Hence, while regular Stata syntax is
also supported, greshape provides the aliases by() and keys()
for i() and j() respectively (note spread and wide
accept multiple j keys).

wide->long

greshape gather
varlist,
keys(varlist)
values(varname)
[options]

greshape long
stubnames,
by(varlist)
[options]

long->wide

greshape spread
varlist,
keys(varname)
[options]

greshape wide
stubnames,
by(varlist)
keys(varname)
[options]

i and j not only look similar, but even if you are
familiar with conventional matrix notation it's not immediately obvious
which is the grouping variable and which defines the column keys. In any
case, note that in spread and gather the by() option is
implicitly defined as every variable remaining in the data.

Stata syntax

I think the above syntax is clearer (happy to receive feedback otherwise)
but greshape also accepts the traditional Stata i, j syntax:

greshape long
stubnames,
i(varlist)
[options]

greshape wide
stubnames,
i(varlist)
j(varname)
[options]

Details

The stubnames are a list of variable prefixes. The suffixes are either
saved or taken from keys(), depending on the shape of the data. Remember
this picture:

long
+
+ wide
| i j stub | +
+
|
| | i stub1 stub2 |
| 1 1 4.1 | greshape |
|
| 1 2 4.5 | <
> | 1 4.1 4.5 |
| 2 1 3.3 | | 2 3.3 3.0 |
| 2 2 3.0 | +
+
+
+

long->wide

j existing variable(s)
/
greshape wide stub, by(i) keys(j)

wide->long

greshape long stub, by(i) keys(j)
\
j new variable

Additionally, the user can reshape in the style of R's tidyr package.
To go from long to wide:

greshape spread varlist, keys(j)

Note that spread (and gather) both require variable names, not prefixes.
Further, all variables not specified in the reshape are assumed to be
part of by() and the new variables are simply named after the values of
keys(). From wide to long:

greshape gather varlist, keys(j) values(values)

This does not check for duplicates or sorts the data. Variables not
named are assumed to be part of by()). The values of the variables in
varlist are saved in values(), with their names saved in keys().

reshape's extended syntax is not supported; that is, greshape does
not implement "reshape mode" where a user can type reshape long or
reshape wide after the first reshape. This syntax is cumbersome to
support and prone to errors given the degree to which greshape had
to rewrite the base code. This also means the "advanced" commands
are not supported, including: clear, error, query, i, j, xij, and xi.

Long
* by(varlist) use varlist as the ID variables (alias i()).

keys(varname) wide->long: varname, new variable to store stub suffixes (default _j; alias j()).

string Whether to allow for string matches to each stub

match(str) Where to match levels of keys() in stub (default @). Use match(regex) for complex matches.

dropmiss Drop missing observations for reshaped variables.

Wide
* by(varlist) use varlist as the ID variables (alias i()).

* keys(varlist) long->wide: varlist, existing variable with stub suffixes (alias j()).

colsepparate(str) Column separator when multiple variables are passed to keys().

match(str) Where to replace levels of keys() in stub (default @).

labelformat(str) Custom label format using placeholders (only with single variable input for key()/j()). Default #keyvalue# #stublabel#. Other placeholders are #stubname#, #keyname#, #keylabel#, #keyvaluelabel#.

prefix(str) Custom renaming of reshaped variables (one rename per stub; @ syntax allowed).

Common long and wide options
fast Do not wrap the reshape in preserve/restore pairs.

unsorted Leave the data unsorted (faster). Original sort order is not preserved.

nodupcheck wide->long, allow duplicate by() values (faster).

nomisscheck long->wide, allow missing values and/or leading blanks in keys() (faster).

nochecks This is equivalent to all 4 of the above options (fastest).

xi(drop) Drop variables not in the reshape, by(), or keys().

Gather
* values(varname) Store values in varname.

keys(varname) wide->long: varname, new variable to store variable names (default _key).

uselabels[(str)] Store variable labels instead of their names (optionally specify which variables to do this for).

dropmiss Drop missing observations for reshaped variables.

Spread
* keys(varlist) long->wide: varlist, existing variable with variable names.

labelformat(str) Custom label format using placeholders (only with single variable input for key()/j()). Default #keyvalue# #stublabel#. Other placeholders are #stubname#, #keyname#, #keylabel#, #keyvaluelabel#.

prefix(str) Custom renaming of reshaped variables (one common rename; @ syntax allowed).

Common gather and spread options
by(varlist) check varlist are the ID variables. Throws an error otherwise.

xi(drop) Drop variables not in the reshape or in by().

fast Do not wrap the reshape in preserve/restore pairs.

Gtools Options
compress Try to compress strL to str#.

forcestrl Skip binary variable check and force gtools to read strL variables.

verbose Print info during function execution.

bench[(int)] Benchmark various steps of the plugin. Optionally specify depth level.

hashmethod(str) Hash method (default, biject, or spooky). Intended for debugging.

oncollision(str) Collision handling (fallback or error). Intended for debugging.

* options are required for that subcommand.

Description

greshape converts data from wide to long form and vice versa. It
is a fast alternative to reshape, and it additionally implements
greshape spread and greshape gather, both of which are
marginally faster and in the style of the equivalent R commands from
tidyr.

It is well-known that reshape is a slow command, and there are
several alternatives that I have encountered to speed up reshape,
including: fastreshape, parallel, sreshape, and various
custom solutions (e.g. here).
While these alternatives are slower than `greshape`, the speed gains are
not uniform.

If j() is numeric, the data is already sorted by i(), and
there are not too may variables to reshape, then fastreshape comes
closest to achieving comparable speeds to greshape. Under most
circumstances, however, greshape is typically 20-60% faster than
fastreshape on sorted data, and up to 90% faster if j() has
string values or if the data is unsorted (by default, greshape will
output the data in the correct sort order). In other words, greshape's
speed gains are very robust, while other solutions' are not.

Examples

See the
online documentation
for examples.

Author

Mauricio Caceres

mauricio.caceres.bravo@gmail.com

mcaceresb.github.io

Website

greshape is maintained as part of the [R] gtools project at github.com/mcaceresb/stata-gtools

Acknowledgment

This help file was based on StataCorp's own help file for reshape.

gtools was largely inspired by Sergio Correia's ftools:
https://github.com/sergiocorreia/ftools.

The OSX version of gtools was implemented with invaluable help from @fbelotti;
see https://github.com/mcaceresb/stata-gtools/issues/11.

Also see

help for
gtools; reshape