help groupfunctionv2.2 28/Jan/2021

Title

groupfunction - Replaces several basic collapse functions (mean, sum, variance, standard deviation, first, max, min). The command is several orders of magnitude faster than collapse when summarizing multiple vectors on larger datasets.}

Syntax

groupfunction [if] [in] [aw pw fw] ,
by(varlist)
mean(varlist)
sum(varlist)
rawsum(varlist)
first(varlist)
min(varlist)
max(varlist)
count(varlist)
sd(varlist)
variance(varlist)
gini(varlist)
theil(varlist)
xtile(varlist)
nq(int)
missing
norestore

Description

groupfunction Replaces several collapse functions (mean, sum, variance, first, max, min). The command is several orders of magnitude faster than collapse

Options

by(varlist) Grouping for reporting estimates.

xtile(varlist) Coupled with nq(), it creates variable with percentiles and adds it to the by() option.

nq(int) Option only works when xtile() is specified. It indicates the number of quantiles.

mean(varlist) Calculates means of specified variables.

sum(varlist) Calculates total sum of specified variables. If weights are specified it will give population expanded total.

rawsum(varlist) Calculates total sum of specified variables. If weights are specified it will ignore weights.

first(varlist) Provides first observation by groups specified in by option.

min(varlist) Provides minimum value, by groups, of vectors specified.

max(varlist) Provides maximum value, by groups, of vectors specified.

count(varlist) Provides observation count, by groups, of vectors specified.

sd(varlist) Calculates standard deviations of specified variables.

variance(varlist) Calculates variance of specified variables.

gini(varlist) Calculates Gini coefficient of specified variables.

theil(varlist) Calculates Theil coefficient of specified variables.

missing This option is only relevant for sum and rawsum. If an entire group in by() is missing for the sum/rawsum variable the output will be missing instead of zero.

norestore Drops all non-relevant variables before calculations to improve memory management.

slow Use this option if you run into memory issues, it will get values one by one.

merge Requests that values are not to be collapsed, it instead merges the new vectors to the dataset in memory.

Example

. sysuse auto, clear

. groupfunction [aw=weight], mean(price) min(weight) by(foreign)

Example 2 (Time comparisons)

clear all
set more off
set obs 300000

version 13
set seed 458267

gen regions = int(runiform()*20)

replace region = 1 if inrange(region,0,3)
replace region = 2 if inrange(region,4,5)

//Income per capita
forval z=1/200{
gen x_`z' = region*runiform()*4000 + rnormal()*200
}

forval z=1/300{
gen y`z' = runiform()*100 + (rnormal()*20)^2
}

gen weight = abs(rnormal())

//time collapse
preserve
timer on 1
collapse (mean) y* x_* [aw=w], by(region)
timer off 1
restore

//time fcollapse (Weights not supported)
preserve
timer on 2
fcollapse (mean) y* x_*, by(region)
timer off 2
restore

//time groupfunction
preserve
timer on 3
groupfunction [aw=weight], mean(y* x_*) by(region)
timer off 3
restore

timer list

Authors

Paul Corral

The World Bank - Poverty and Equity Global Practice

Washington, DC

pcorralrodas@worldbank.org

Minh C. Nguyen

The World Bank - Poverty and Equity Global Practice

Washington, DC

pcorralrodas@worldbank.org

Joao Pedro Azevedo

The World Bank - Poverty and Equity Global Practice

Washington, DC

jazevedo@worldbank.org

Raul Andrés Castañeda provided valuable suggestions for this command.