*! version 1.3.74 29Apr2018
help wgtcellcollapse (Stas Kolenikov's webpage)

Title

wgtcellcollapse -- Collapsing weighting cells

Syntax

wgtcellcollapse task
[if] [in]
[, options]

where task is one of:

define define collapsing rules explicitly

sequence create collapsing rules for a sequence of categories

report list the currently defined collapsing rules

candidate find rules applicable to a given category

collapse perform cell collapsing

label label collapsed cells using the original labels after wgtcellcollapse collapse

Description

wgtcellcollapse collapses cells defined by categorical variable(s),
i.e., performs a series of recode operations combining cells with small counts
to ensure that a minimal cell size is achieved in every resulting cell.
Typically, the need for this task arises with nonreponse or calibration adjustments
to produce survey weights.

As the syntax of the individual subcommands varies wildly, the individual syntax and options
are spelled out under the corresponding tasks, and mini-examples of the code are provided, as well.

Syntax of wgtcellcollapse define

wgtcellcollapse define, variables(varlist) [options]

wgtcellcollapse define defines individual collapsing rules. It is typically used
to handle edge cases that automated rules
(defined by wgtcellcollapse sequence)
have difficulties identifying.

define task options

variables(varlist) (required) specifies the list of variables for which the collapsing rule can be used

from(numlist) specifies the list of source categories that can be collapsed according to this rule

to(#) specifies the numeric value of the new, collapsed category

label(text) provides the value label to be attached to the new, collapsed category

max(#) overrides the automatically determined max value of the collapsed variable

clear clears all the rules currently defined

Individual collapsing rules can be defined as follows.

. clear

. set obs 4

. gen byte x = _n

. label define x_lbl 1 "One" 2 "Two" 3 "Three" 4 "Four"

. label values x x_lbl

. wgtcellcollapse define, var(x) from(1 2 3) to(123)

. wgtcellcollapse report, var(x)

Rule (1): collapse together

x == 1 (One)

x == 2 (Two)

x == 3 (Three)

into x == 123 (123)

WARNING: unlabeled value x == 123

The user would need to develop convenient mnemonic rules for the numeric values
of the collapsed categories that make it easy to understand what the underlying
source cells are. In the example above, the value of the collapsed cell, 123,
reads like "1, 2 and 3", making it easier to identify that the applicable rule
had to do with collapsing cells 1, 2 and 3. Some other mnemonic, like 10203,
padding the individual categories with zeroes, or 10003, indicating a stretch
of all categories between 1 and 3, might also have made sense.
If the underlying collapsing rules are more complex, and/or the source categories
have more digits, the mnemonic values of the collapsed categories may have to
be simplified. See below how wgtcellcollapse sequence
deals with the issue.

Syntax of wgtcellcollapse sequence

wgtcellcollapse sequence, variables(varlist) from(numlist) depth(#)

wgtcellcollapse sequence automatically creates rules that allow collapsing subsets
of a series of categories of the source variable(s).

sequence task options

variables(varlist) specifies the list of variables for which the collapsing rule can be used

from(numlist) specifies the sequence of values from which the plausible subsequences can be constructed

depth(#) specifies the maximum number of the source categories that can be collapsed (length of subsequence to be collapsed)

All three options are required.

Moderate length sequences of collapsing categories can be defined as follows.

. clear

. set obs 4

. gen byte x = _n

. label define x_lbl 1 "One" 2 "Two" 3 "Three" 4 "Four"

. label values x x_lbl

. wgtcellcollapse sequence, var(x) from(1 2 3 4) depth(3)

. wgtcellcollapse report, var(x)

Rule (1): collapse together

x == 1 (One)

x == 2 (Two)

into x == 212 (One to Two)

Rule (2): collapse together

x == 2 (Two)

x == 3 (Three)

into x == 223 (Two to Three)

...

Rule (8): collapse together

x == 2 (Two)

x == 234 (Three to Four)

into x == 324 (Two to Four)

Rule (9): collapse together

x == 4 (Four)

x == 223 (Two to Three)

into x == 324 (Two to Four)

When creating sequential collapses, wgtcellcollapse sequence uses the following mnemonics
in defining the new categories:

* The first digit or two contain the length of the collapsed subsequence (up to depth(#)).

* Then comes the starting value of the category in the subsequence (padded by zeroes as needed).

* Then comes the ending value of the category in the subsequence (padded by zeroes as needed).

In the example above, rules 7 through 9 lead to collapsing into the new category 324. This
should be interpreted as ``the subsequence of length 3 that starts with category 2 and ends with category 4''.
A numeric value of the collapsed category that reads like 50412 means
``the subsequence of length 5 that starts with category 4 and ends with category 12'',
where wgtcellcollapse sequence padded the value of 4 with an additional zero,
so that the length of resulting collapsed category value is always (# of digits of the sequence length) +
twice (# of digits of the greatest source category).

Note that wgtcellcollapse sequence automatically created labels for the collapsed cells.

The order of the categories in the from() option matters: wgtcellcollapse sequence respects
that order, and does not sort the values. If the categories are supplied
in the order 2, 4, 1 and 3, then wgtcellcollapse sequence would collapse 2 with 4, 4 with 1,
and 1 with 3:

. wgtcellcollapse define, var(x) clear

. wgtcellcollapse sequence, var(x) from(2 4 1 3) depth(2)

. wgtcellcollapse report, var(x)

Rule (1): collapse together

x == 2 (Two)

x == 4 (Four)

into x == 224 (Two to Four)

Rule (2): collapse together

x == 4 (Four)

x == 1 (One)

into x == 241 (Four to One)

Rule (3): collapse together

x == 1 (One)

x == 3 (Three)

into x == 213 (One to Three)

Syntax of wgtcellcollapse report

wgtcellcollapse report, variables(#) [ break ]

wgtcellcollapse report reports the existing rules, if any are defined for the variable(s) of interest.

report task options

variables(varlist) (required) specifies the list of variables for which the collapsing rules are to be reported

break makes wgtcellcollapse exit with error when technical inconsistencies are encountered

The intent of the break option of wgtcellcollapse report is to act as a quality control
measure and abort the execution
when technical deficiencies in the rules or in the data are encountered:

. clear

. set obs 4

. gen byte x = _n

. label define x_lbl 1 "One" 2 "Two" 3 "Three" 4 "Four"

. label values x x_lbl

. wgtcellcollapse define, var(x) from(1 2 3) to(123)

. wgtcellcollapse report, var(x) break

Rule (1): collapse together

x == 1 (One)

x == 2 (Two)

x == 3 (Three)

into x == 123 (123)

ERROR: unlabeled value x == 123

assertion is false

r(9);

In this case,
the label of the new category 123 was not defined, and this is considered a serious
enough deficiency to stop.

Syntax of wgtcellcollapse candidate

wgtcellcollapse candidate, variable(varname) category(#) [maxcategory(#)]

wgtcellcollapse candidate finds the rules that can be used
to collapse a given category of the source variable.

candidate task options

variable(varname) (required) the variable to be collapsed

category(#) (required) the category to be collapsed

maxcategory(#) the maximum value of the categories in the candidate rules to be returned

The rules found are quietly returned through the mechanism of sreturn,
see return, as they are intended to stay in memory sufficiently long for
wgtcellcollapse collapse to evaluate each rule.

. wgtcellcollapse candidate, var(x) cat(2)

. sreturn list

s(goodrule) : "1 2 4 7 8"

s(rule8) : "2:234=324"

s(rule7) : "2:3:4=324"

s(rule4) : "1:2:3=313"

s(rule2) : "2:3=223"

s(rule1) : "1:2=212"

s(cat) : "2"

s(x) : "x"

If we don't want to use the cells that have already been collapsed,
like 234 in the example above, the collapsing rules can be restricted
with max(#) option (assuming that mnemonics for collapsed cells provide
longer labels for collapsed cells, as is done by wgtcellcollapse sequence):

. wgtcellcollapse candidate, var(x) cat(2) max(9)

. sreturn list

s(goodrule) : "1 2 4 7"

s(rule7) : "2:3:4=324"

s(rule4) : "1:2:3=313"

s(rule2) : "2:3=223"

s(rule1) : "1:2=212"

s(cat) : "2"

s(x) : "x"

Requests for nonexisting categories are not considered errors, but simply produce empty lists
of ``good rules''.

. . wgtcellcollapse candidate, var(x) cat(55)

. sreturn list

s(cat) : "55"

s(x) : "x"

Syntax of wgtcellcollapse collapse

wgtcellcollapse collapse [if] [in],
variables(varlist)
mincellsize(#)
saving(dofile_name)
[ options ]

wgtcellcollapse collapse provides the central target functionality of the program,
i.e., processing the sample data to collapse weighting cells.

Required collapse task options

variables(varlist) provides the list of variables whose cells are to be collapsed.
When more than one variable is specified, wgtcellcollapse collapse proceeds from right to left,
i.e., first attempts to collapse the rightmost variable.

mincellsize(#) specifies the minimum cell size for the collapsed cells. For most weighting
purposes, values of 30 to 50 can be recommended.

saving(dofile_name) specifies the name of the do-file that will contain the cell collapsing code.

do-file management collapse task options

replace overwrites the do-file if one exists.

append appends the code to the existing do-file.

run specifies that the do-file created is run upon completion. This option is typically specified with most runs.

Other collapse task options

generate(varname) specifies the name of the collapsed variable to be created.

feed(varname) provides the name of an already existing collapsed variable.

strict modifies the behavior of wgtcellcollapse collapse so that only
collapsing rules for which all participating categories have nonzero counts are utilized.

sort(varlist) sorts the data set before proceeding to collapse the cell.
The default sort order is in terms of the values of the collapsed variable.
A different sort order may produce a different set of collapsed cell when
cells are tied on size.

maxpass(#) specifies the maximum number of passes through the data set. The default value is 10000.

maxcategory(#) is the maximum category value of the variable being collapsed.
It is passed to internal calls to wgtcellcollapse candidate, ... maxcategory(), see
wgtcellcollapse candidate.

zeroes(numlist) provides a list of the categories of the collapsed
variable that may have zero counts in the data.

greedy modifies the behavior wgtcellcollapse collapse to prefer the rules
that collapse the maximum number of categories.

The primary intent of wgtcellcollapse collapse is to create the code that can be
utilized for both the survey data file and the population targets data file, provided that
the variables have identical names and categories in these two files.
Thus it does not only manipulate the data in the memory
and collapses the cells, but also produces the do-file code that can be recycled.
To that effect, when a do-file is created with the replace and saving() options,
the user needs to specify generate() option to provide the name of the collapsed variable;
and when the said do-file is appended with the the append and saving() options,
the name of that variable is provided with the feed() option.

The algorithm wgtcellcollapse collapse uses to identify the cells to be collapsed is
a variation of greedy search.
It first identifies the cells with the lowest (positive) counts; finds the candidate rules
for the variable(s) to be collapsed; evaluates the counts of the collapsed cells across all
these candidate rules; and uses the rule that produces the smallest size of the
collapsed cell across all applicable rules. So when it finds several rules that are applicable
to the cell being currently processed that has a size of 5, and the candidate rules produce cells
of sizes 7, 10 and 15, wgtcellcollapse collapse will use the rule that produces the cell
of size 7. The algorithm runs until all cells have sizes of at least
mincellsize(#) or until maxpass(#) passes through the data are executed.
It is a pretty dumb algorithm, actually, and it fails quite often.
For that reason, a number of hooks are provided to modify its behavior.
Section \refexample will demonstrate the typical failures, and the ways to overcome them.

Hint 1. Since wgtcellcollapse collapse works with the sample data,
it will not be able to identify categories that are not observed in the sample (e.g., rare categories),
but may be present in the population. This will lead to errors at the raking stage,
when the control total matrices have more categories than the data, forcing ipfraking to stop
with errors.
To help with that, the option zeroes() allows the user to pass the categories
of the variables that are known to exist in the population but not in the sample. That
way, wgtcellcollapse collapse can produce the collapsing code for those categories;
it has zero effect in the sample, but touches upon these categories in the population.

Hint 2. The behavior of wgtcellcollapse collapse, zeroes() may still not be
satisfactory. As it evaluates the sample sizes of the collapsed cells across a number
of candidate rules that involve zero cells, it may pick up a rule with the fewest categories involved,
and that rule may as well leave some other candidate rules with zero cells untouched.
This may create problems when wgtcellcollapse collapse returns to those untouched cells,
and looks for the existing cells to collapse them with, creating collapsing rules with breaks
in the sequences. To improve upon that behavior, option greedy makes
wgtcellcollapse collapse look for a rule that has the greatest number of elements, thus collapsing
as many categories with zero counts in one swipe as it can.

Hint 3. Other than for dealing with zero cells, the option strict should be specified
most of the times. It makes sure that the each element in the candidate rules correspond to the actual data
and has a current non-zero count in the sample.

Hint 4. Sometimes, you see some combinations in the data that seem like a nobrainer
to collapse. Well, they are nobrainers to you, but wgtcellcollapse collapse is not that smart.
If you want to guarantee some specific combination of cells to be collapsed by wgtcellcollapse collapse,
your best bet may be to explicitly identify them with the if conditions, and specify some
ridiculously large cell size like mincellsize(10000) so that wgtcellcollapse collapse makes every possible
effort to collapse those cells. It will exit with a complaint that this size could not be achieved,
but hopefully the cells will be collapsed as needed.

Hint 5. The standard use case of wgtcellcollapse involves using several variables
and creating interactions of two or more categorical variables, where one of the variables is highly detailed
(say dozens of categories in population) while the other is less detailed (say a handful of categories in the population).
If only one variable needs to be collapsed, you would need to provide an extra dummy variable to
wgtcellcollapse collapse as a placeholder.

Continuing with the example with four categories, this minimal example collapses
two smallest cells:

. expand x

. gen byte _one = 1

. label variable _one "Dummy"

. wgtcellcollapse define, var(_one) from(1 2) to(12) label("One, two")

. wgtcellcollapse collapse, var(_one x) min(3) gen(cx) saving(collapse_x.do) replace run

Pass 0 through the data...

smallest count = 1 in the cell 1001

Invoking rule 1:2=212

replace cx = 1212 if inlist(cx, 1001, 1002)

Pass 1 through the data...

smallest count = 3 in the cell 1003

Done collapsing! Exiting...

. *** Automatically created on 13 Dec 2017 at 22:45:37

. * Source syntax: wgtcellcollapse collapse , var(_one x) min(3) gen(cx) saving(collapse_x.do) replace

. generate long cx = (_one)*1000 + x

. label variable cx "Long ID of the interaction"

. format cx %12.0f

. char cx[sources] _one x

. char cx[max] 12324

. replace cx = 1212 if inlist(cx, 1001, 1002)

(3 real changes made)

. char cx[rule1] 1001:1002=1212

. char cx[nrules] 1

.

end of do-file

Here, wgtcellcollapse collapse:

1. figured out that the greatest value of x it can reasonably expect, based on all of the collapsing
rules that x is endowed with, is 324, so the interaction in question should leave space for three decimal
places for the collapsed values of x;

2. created an interaction of the source variable x with the dummy variable _one;

3. provided some relevant metadata for the interaction variable cx;

4. identified the cell with the smallest size, cx == 1001;

5. (no output) looked at all the available rules to collapse x==1;

6. among these rules, found the one with the smallest resulting count (collapse x==1 with x==2);

7. recalibrated the intermediate category counts;

8. compared the recalibrated counts with the target given by min(3) option;

9. concluded that no further collapsing is necessary; and

10. ran the resulting do-file that was produced.

Syntax of wgtcellcollapse label

wgtcellcollapse label, variable(varname),
[ verbose force ]

wgtcellcollapse label generates labels for the collapsed data.

label task options

variable(varname) (required) is the collapsed variable to be labeled.

verbose outputs the labeling results. There may be a lot of output.

force instructs wgtcellcollapse label to only use categories present in the data.

Utilizing the multiple language functionality of label_language,
wgtcellcollapse label automatically creates labels based on the collapsed categories.

. wgtcellcollapse label, var(cx)

WARNING: source variable _one is not labeled

(language default renamed unlabeled_ccells)

(language numbered_ccells now current language)

(language texted_ccells now current language)

To attach the numeric labels (of the kind "cx==1003"), type:

label language numbered_ccells

To attach the text labels (of the kind "cx==1; Three"), type:

label language texted_ccells

The original state, which is also the current state, is:

label language unlabeled_ccells

. label language numbered_ccells

. tab cx

Interactions of |

_one x, with |

some collapsing | Freq. Percent Cum.

+

_one==1, x==3 | 3 30.00 30.00

_one==1, x==4 | 4 40.00 70.00

_one==1, x==212 | 3 30.00 100.00

+

Total | 10 100.00

. label language texted_ccells

. tab cx

Interactions of |

_one x, with |

some collapsing | Freq. Percent Cum.

+

1; Three | 3 30.00 30.00

1; Four | 4 40.00 70.00

1; One to Two | 3 30.00 100.00

+

Total | 10 100.00

. label language unlabeled_ccells

. tab cx

Interactions of |

_one x, with |

some collapsing | Freq. Percent Cum.

+

1003 | 3 30.00 30.00

1004 | 4 40.00 70.00

1212 | 3 30.00 100.00

+

Total | 10 100.00

References

Kolenikov, S. (2014). Calibrating survey data using iterative proportional fitting.
The Stata Journal, 14 (1), pp. 22--59.
Stata Journal website

Kolenikov, S. (2018). Updates to the ipfraking ecosystem.
Author's website

Author

Stanislav Kolenikov

Senior Scientist

Abt Associates

skolenik at gmail dot com

Also see

ipfraking -- weight calibration using iterative proportional fitting