help for expgen and lexpgen(Roger Newson)

Duplicate observations sorted in original order with generated variables

expgen [ [ newvarname1 ] = ] ncexp [if] [in] [ ,
oldseq(newvarname2)
copyseq(newvarname3)
sortedby(sortedby_interpretation)
order
missing zero
fast
]

lexpgen [ newvarname1 ] [if] [in] , levels(numlist) [ clevels(newvarname4)
expgen_options
]

where sortedby_interpretation is one of

ignore | group | unique

or any abbreviation of any of these,
and expgen_options are any options used by expgen.

Description

expgen is an extended version of expand.
It replaces each observation in the
current dataset with nc copies of the observation, sorted in the original order,
where nc is equal to the integer part int(ncexp) of the required expression ncexp.
expgen may also add new variables.
These are newvarname1 (containing the value of the expression ncexp),
newvarname2 (containing the sequential order of the original observation in the old dataset),
and newvarname3 (containing the sequential order of each copy in the set of copies
generated from its original observation).
If missing and/or zero is specified,
then observations in the old dataset with missing and/or non-positive values
for int(ncexp) are each replaced by one observation in the new dataset.
lexpgen is an alternative version of expgen.
It replaces each observation in the
current dataset with one copy of the observation for each of a user-specified input list of numbers,
sorted primarily in the original order,
and secondarily by the numbers in the list.
It may add the new variables added by expgen,
and/or another new variable newvarname4,
containing the number in the input list of numbers corresponding to the copy.

Options for expgen and lexpgen

oldseq(newvarname2) specifies a new variable to be generated, containing the sequential
order of the observation in the old dataset from which the current
observation was copied, as defined in the old dataset.
This old sequential order is defined after the exclusion from the old dataset of observations
not satisfying the if and in requirements, and after the exclusion of
observations with missing values of ncexp (if missing was not specified)
and of observations with non-positive values of int(ncexp)
(if zero was not specified).
(See missing and zero below.)

copyseq(newvarname3) specifies a new variable to be generated, containing the sequential
order of the copied observation in the new dataset, in the set of
observations copied from the same original observation in the old dataset.
If an observation in the old dataset has a value int(ncexp) equal to k,
then the k duplicates of that observation in the new dataset have values
of the copyseq() variable in ascending order from 1 to k.

sortedby(sortedby_interpretation) specifies the interpretation given to the varlist
by which the input dataset is sorted,
given as output by the extended macro function sortedby.
This option is used to specify the varlist by which the output dataset will be sorted.
If sortedby(ignore) is specified,
then the varlist by which the input dataset is sorted is ignored,
and the output dataset is sorted primarily by the oldseq() variable (if specified),
and secondarily by the copyseq() variable (if specified).
If sortedby(group) is specified,
then the varlist by which the input dataset is sorted is assumed to be a group identifier,
possibly with multiple observations with the same combination of values,
and the output dataset is sorted primarily by these variables,
and secondly by the oldseq() variable (if specified),
and thirdly by the copyseq() variable (if specified).
If sortedby(unique) is specified,
then the varlist by which the input dataset is sorted is assumed to be a unique observation identifier,
with no more than one observation with the same combination of values
in the subset of observations specified by the in and if qualifiers,
and the output dataset is sorted primarily by these variables,
and secondly by the copyseq() variable (if specified).
If the user specifies sortedby(unique)
and the dataset is in fact not sorted,
or is in fact sorted by variables which do not uniquely identify the observations,
then expgen fails with an error message.
If the user does not specify a sortedby() option,
then sortedby(ignore) is assumed.
Note that, whichever sortedby() option is specified,
the order of the observations in the output dataset will be the same,
and will be primarily the order of the observations in the input dataset from which the copies were made,
and secondarily by the copyseq() variable (if specified),
or equivalently by the clevel() variable (if specified by lexpgen).
The sortedby() option only affects the list of variables recognized by Stata
as defining the sorting order,
as reported by describe.

order specifies that the variables by which the output dataset is sorted
will be reordered to the start of the variable order.
These variables are as specified by the sortedby() option.
If order is specified,
then they will be the first variables in the output dataset listed by describe.

missing specifies that observations in the old dataset with missing values of
ncexp are each to be replaced in the new dataset with a single observation.
If missing is absent, then such old observations are deleted from the new dataset.

zero specifies that observations in the old dataset with non-positive values
of int(ncexp) are each to be replaced in the new dataset with a single observation.
If zero is absent, then such old observations are deleted from the new dataset.

fast is an option for programmers.
It specifies that expgen and lexpgen will take no action to restore the original dataset
if expgen or lexpgen fails,
or if the user presses Break.

Options for lexpgen only

levels(numlist) is compulsory.
It specifies an input list of numeric levels
which lexpgen will use to create an expanded dataset,
with 1 observation per number in the list
per original observation in the unexpanded dataset.
Note that the input list may contain numeric missing values,
but repeated values are removed.

clevel(newvarname) specifies the name of a new numeric variable,
containing, in each observation in the expanded dataset,
the corresponding number in the list of numbers specified by the levels() option.
Observations in the new dataset will then be multiple copies of observations in the old unexpanded dataset,
sorted
primarily by the position of the corresponding observation in the old unexpanded dataset,
and secondarily by the value of the new variable specified by clevels().

Remarks

expgen is designed as an improved version of expand.
Unlike expand, it returns duplicate observations sorted in the original order of the
corresponding observations in the old dataset, from which they were copied.
There is also the option of adding new variables to the duplicate observations.
expgen was designed for use, in a medical setting, in the case when the
original dataset is from a spreadsheet with one row per patient, and with
multiple columns containing multiple repeated measurements on the same patient
at different times and/or on different parts of the same patient's body.
We might want to expand this dataset to a new dataset with 1 observation per
repeated measurement, and then perform a repeated measures analysis.
The copyseq() variable can be used to generate a new variable in this new dataset,
containing the repeated measures.
The new variable might have values each copied from one of a list of variables in the original dataset,
depending on the value of the copyseq() variable.
(See Examples.
In SAS, this kind of duplication is done using a SAS data step
containing a SAS OUTPUT statement inside a SAS DO-loop,
iterating over a list of variables defined as a SAS array.)

expgen can be used with the keyby package, downloadable from SSC,
to enforce the relational database model in Stata.
In the relational database model, a dataset is viewed as a mathematical function,
whose domain is the set of all available value combinations of a list of key variables,
and whose range is the set of all possible value combinations of the non-key variables.
If a dataset is keyed using keyby,
and is then expanded using expgen,
specifying the copyseq() option and sortedby(unique),
then the expanded dataset generated by expgen
will automatically be a relational dataset,
keyed primarily by the original key variables specified by keyby,
and secondly by the copyseq() variable.
And, if the user also specifies the order option with expgen,
then the key variables will be the first in the order of variables in the output dataset.

Examples

. expgen nreps=rep78, old(modseq) copy(repseq) miss zero

. sysuse auto, clear

. sort foreign make

. expgen nreps=rep78, copy(repseq) sortedby(unique) order

. describe

The following example uses the census dataset, and expands it from a dataset
of US states to a dataset of combinations of US state and age group:

. sysuse census, clear

. expgen =4, copy(agegp)

. lab def agegp 1 "0-4" 2 "5-17" 3 "18-64" 4 "65+"

. lab val agegp agegp

. gene popul = (agegp==1)*poplt5 + (agegp==2)*pop5_17 + (agegp==3)*(pop18p-pop65p) + (agegp==4)*pop65p

. drop pop poplt5 pop5_17 pop18p pop65p

. desc

. list state agegp popul

The following example uses the housing dataset,
and uses lexpgen to expand it from a dataset
of US states to a dataset of combinations of US state and current year.
Note that the option sortedby(unique)
specifies that the variable state identifies the input observations uniquely,
and ensures that the output dataset is identified as being sorted by state and year:

. use http://www.stata-press.com/data/r9/hsng.dta, clear

. desc

. gene pop70=pop*100/(100+popgrow)

. lab var pop70 "Population in 1970"

. lexpgen, levels(1970 1980) clevel(year) sortedby(unique)

. gene popul = pop70*(year==1970) + pop*(year==1980)

. format popul %10.1f

. lab var year "Current year"

. lab var popul "Current population"

. desc, fu

. list state year popul, sepby(state)

Author

Roger Newson, Queen Mary University London, UK.

Email: r.newson@qmul.ac.uk

Also see

Manual: [D] expand, [D] expandcl, [D] joinby, [D] contract, [D] collapse, [D] reshape,
[D] order, [D] describe

On-line: help for expand, expandcl, joinby, contract, collapse, reshape, order, describe

help for keyby, expandby if installed