*! version 0.1.0 14Mar2022

Title

gstats hdfe -- Absorb HDFE (residualize variables)

Important: Please run gtools, upgrade to update gtools to
the latest stable version.

Warning: gstats hdfe is in beta; see missing features.
(To enable beta, define global GTOOLS_BETA = 1.)

Syntax

gstats hdfe
varlist
[if] [in]
[weight]
[
, absorb(varlist)
{gen(newvarlist)|prefix(str)|replace}
options
]

If none of gen(), prefix(), or replace are
specified then target=source syntax must be supplied
instead of varlist:

target_var=varname
[target_var=varname ...]

gstats hdfe (alias gstats residualize) provides a fast way of
absorbing high-dimensional fixed effects (HDFE). It saves the number of levels
in each absorbed variable, accepts weights, and optionally takes by()
as an argument (in this case ancillary information is not saved by
default and must be accessed via mata()). Missing values in the
source and absorb variables are skipped row-size (the latter can be
optionally retained via absorbmissing).

Specify Targets
prefix(str) Generate all variables with prefix (e.g. residualized x saved to prefix_x, etc).

generate(newvarlist) List of targets; must specify one per source.

replace Replace variables as applicable. (If no targets are specified, this replaces the sources.)

wildparse Allow rename-style syntax if target=source is specified (e.g. x*=prefix_x*).

HDFE Options
by(varlist) Group by variables.

matasave[(str)] Save by() info (and absorb info by group) in mata object (default name is GtoolsByLevels)

absorbmissing Treat missing absorb levels as a group instead of dropping them.

algorithm(str) Algorithm used to absorb HDFE: CG (conjugate gradient), MAP (alternating projections), SQUAREM (squared extrapolation), IT (Irons and Tuck).

maxiter(int) Maximum number of algorithm iterations (default 100,000). Pass . for unlimited iterations.

tolerance(real) Convergence tolerance (default 1e-8).

traceiter Trace algorithm iterations.

standardize Standardize variables before algorithm.

Gtools Options
compress Try to compress strL by() variables to str#.

forcestrl Skip binary by() variables check and force gtools to read strL by() variables.

verbose Print info during function execution.

bench[(int)] Benchmark various steps of the plugin. Optionally specify depth level.

hashmethod(str) Hash method for by() variables (default, biject, or spooky). Intended for debugging.

oncollision(str) Collision handling (fallback or error). Intended for debugging.

aweights, fweights, and pweights are
allowed (see [U] 11.1.6 weight for more on the way Stata
uses weights).

Description

gstats hdfe (alias gstats residualize) is designed as a
utility to embed in programs that require absorbing high-dimensional
fixed effects, optionally taking in weights. The number of non-missing
observations and the number of levels in each absorb variable are
returned (see stored results).

Mainly as a side-effect of being a gtools program, by() is
also allowed. In this case, the fixed effects are absorbed sepparately
for each group defined by by(). Note in this case the number of
non-missing observations and the number of absorb levels varies by group.
This is NOT saved by default. The user can optionally specify
matasave[(str)] to save information on the by levels,
including the number of non-missing rows per level and the number of
levels per absorb variable per level.

matasave[(str)] by default is stored in
GtoolsByLevels but the user may specify any name desired.
Run mata GtoolsByLevels.desc() for details on the stored
objects (also see stored results below).

Examples

See the
online documentation
for examples.

Stored results

gstats hdfe stores the following in r():

Macros

r(algorithm) algorithm used for HDFE absorption

Scalars

r(N) number of non-missing observations

r(J) number of by() groups

r(minJ) largest by() group size

r(maxJ) smallest by() group size

r(iter) (without by()) iterations of absorption algorithm

r(feval) (without by()) function evaluations in absorption algorithm

Matrices

r(nabsorb) (without by()) vector with number of levels in each absorb variable

With matasave[(str)], the following data is
stored in the mata object:

string matrix nj
non-missing observations in each -by- group

string matrix njabsorb
number of absorbed levels in each -by- group by each absorb variable

real scalar anynum
1: any numeric by variables; 0: all string by variables

real scalar anychar
1: any string by variables; 0: all numeric by variables

string rowvector byvars
by variable names

real scalar kby
number of by variables

real scalar rowbytes
number of bytes in one row of the internal by variable matrix

real scalar J
number of levels

real matrix numx
numeric by variables

string matrix charx
string by variables

real scalar knum
number of numeric by variables

real scalar kchar
number of string by variables

real rowvector lens
> 0: length of string by variables; <= 0: internal code for numeric variables

real rowvector map
map from index to numx and charx

real rowvector charpos
position of kth character variable

string matrix printed
formatted (printf-ed) variable levels (not with option -silent-)

Missing Features

Check whether it's mathematically OK to apply SQUAREM. In general it's meant
for contractions but my understanding is that it can be applied to any
monotonically convergent algorithm.

Improve convergence criterion. Current criterion may not be sensible.

Author

Mauricio Caceres

mauricio.caceres.bravo@gmail.com

mcaceresb.github.io

Website

gstats is maintained as part of the [R] gtools project at github.com/mcaceresb/stata-gtools

Acknowledgment

gtools was largely inspired by Sergio Correia's ftools:
https://github.com/sergiocorreia/ftools, and this specific
function was inspired by Sergio Correia's reghdfe:
https: //github.com/sergiocorreia/reghdfe.

The OSX version of gtools was implemented with invaluable help from @fbelotti;
see https://github.com/mcaceresb/stata-gtools/issues/11.

References

See
online documentation
for the list of references.

Also see

help for
gtools