help for lastupto(Roger Newson)

Condense a dataset to have 1 observation for each of a list of X-values.

lastupto xvarname [if exp]
[in range] [,

xvalues(numlist by(varlist)

list( [varlist] [if exp] [in range] [ , [list_options] ] )

saving(filename[,replace])

norestore fast
flist(global_macro_name)

idnum(#) nidnum(newvarname)
idstr(string) nidstr(newvarname)

format(varlist_1 format_1 ... varlist_n format_n)

noorder keep(varlist)
]

Description

lastupto inputs a dataset in memory with an X-variable, a list of X-values,
and (optionally) a list of by-variables.
It creates an output dataset (or resultsset},
with 1 observation per provided X-value in the list
(or per X-value per by-group),
and data on the variable values in the last observation in the dataset (or by-group)
with an X-value up to and including the provided X-value,
when the observations have been sorted by by-group, then by their X-values,
and then by their pre-existing order.
lastupto is very useful for condensing Kaplan-Meier curves in big datasets.
lastupto uses the SSC package expgen,
which must be installed if lastupto is to work.

Options for use with lastupto

The options are listed in the following 3 groups:

1. Basic options.
(These specify the X-values and by-groups.)

2. Output-destination options.
(These specify where the output dataset will be written.)

3. Output-modifying options.
(These specify additional variables for the output dataset,
or modifications to variables.)

Basic options

xvalues(numlist) specifies a list of numeric input X-values.
The output dataset will have 1 observation per input X-value
(or per X-value per by-group if a by() option is specified),
and data on the values of non-X-variables variables in the last observation in the input dataset
with an X-value up to or including the input X-value.

by(varlist) specifies the by-groups present in the input dataset.
If a by() option is absent, then the output dataset will have 1 observation per input X-value.
If a by() option is present,
then the output dataset will have 1 observation per input X-value per by-group.

Output-destination options

list(varlist [if exp] [in range] [, list_options ] )
specifies a list of variables in the output
dataset, which will be listed to the Stata log by lastupto.
The list() option can be used with the format() option (see below)
to produce a list of summary statistics
with user-specified numbers of decimal places or significant figures.
The user may optionally also specify if or in qualifiers to list subsets of combinations
of variable values,
or change the display style using a list of list_options allowed as options by the list command.

saving(filename[,replace]) saves the output dataset to a disk file.
If replace is specified, and a file of that name already exists,
then the old file is overwritten.

norestore specifies that the output dataset will be written to the memory,
overwriting any pre-existing dataset. This option is automatically set if fast is
specified. Otherwise, if norestore is not specified, then the pre-existing dataset is restored
in the memory after the execution of lastupto.

fast is a stronger version of norestore, intended for use by programmers.
It specifies that the pre-existing dataset in the memory will not be restored,
even if the user presses Break during the execution of lastupto.
If norestore is specified and fast is absent,
then lastupto will go to extra work so that
it can restore the original data if the user presses Break.

Note that the user must specify at least one of the four options list(), saving(),
norestore and fast. These four options specify whether the output dataset
is listed to the Stata log, saved to a disk file, or written to the memory
(overwriting any pre-existing dataset). More than one of these options can be specified.

flist(global_macro_name) specifies the name of a global macro, containing
a filename list (possibly empty). If saving() is also specified, then
lastupto will append the name of the dataset specified in the
saving() option to the value of the global macro specified in flist(). This
enables the user to build a list of filenames in a global macro, containing the
output of a sequence of output datasets.
These files may later be concatenated using append.

Output-modifying options

idnum(#) specifies an ID number for the output dataset.
It is used to create a numeric variable, with default name idnum, in the output dataset,
with that value for all observations.
This is useful if the output dataset is concatenated with other lastupto output datasets
using append, or using dsconcat if installed.

nidnum(newvarname) specifies a name for the numeric ID variable
evaluated by idnum(). If idnum() is present and nidnum() is absent,
then the name of the numeric ID variable is set to idnum.

idstr(string) specifies an ID string for the output dataset.
It is used to create a string variable, with default name idstr in the output dataset,
with that value for all observations.
This is useful if the output dataset is concatenated with other lastupto output datasets
using append, or using dsconcat if installed.

nidstr(newvarname) specifies a name for the string ID variable
evaluated by idstr(). If idstr() is present and nidstr() is absent,
then the name of the string ID variable is set to idstr.

format(varlist_1 format_1 ... varlist_n format_n)
specifies a list of pairs of variable lists and display formats.
The formats will be allocated to
the variables in the output dataset specified by the corresponding varlist_i
lists.

noorder specifies that the by() variables and the X-variable
are not reordered to the beginning of the variable order of the output dataset.
If noorder is not specified,
then the by() variables and the X-variable
are reordered to the beginning of the variable order (see order).

keep(varlist) specifies a list of variables to be kept in the output dataset.

Remarks

lastupto is designed to condense a large dataset
to a dataset small enough to be plotted and/or tabulated.
It is designed for use when a large dataset contains one or more variables
that are defined as a right-continuous function of the X-variable.
This will be the case if the X-variable is a survival time
and other variables include a Kaplan-Meier survival probability,
or if the X-variable is a test score
and the other variables include a sensitivity and/or a specificity.
In these cases, the user might want a dataset with 1 observation for each of a list of X-values,
or with 1 observation per listed X-value per by-group,
and data on sensitivities and/or specificities and/or survival probabilities
and/or their confidence limits.
Such a condensed dataset might be used to produce files contining plots and/or tables
of a manageable size.

lastupto can be very useful for condensing Kaplan-Meier curves in big datasets,
which may have millions of observations,
each with its survival probability and confidence limits,
and therefore the potential to generate gigabyte-sized graphics files
if the dataset is not first condensed to a smaller size with a smaller list of X-values.

To find out more about output-destination and output-modifying options
for resultsset-generating programs,
see the online help for other resultsset-generating packages,
such as the SSC packages
xcollapse, xcontract, descsave and parmest.

Examples

Set-up:

. webuse stan3, clear

. describe, full

. st

. tab posttran, miss

Generate and list survival curves with confidence limits for whole dataset:

. sts gene surv=s lbsurv=lb(s) ubsurv=ub(s)

. describe *surv, full

. lastupto _t, xval(0(50)1000) list(, abbr(32))

. lastupto _t, xval(0(100)1000) keep(_t surv lbsurv ubsurv) list(, abbr(32))

Generate and plot survival curves for by-groups
(using the SSC command eclplot to create confidence-interval plots):

. sts gene surv2=s lbsurv2=lb(s) ubsurv2=ub(s)

. describe *surv2, full

. preserve

. lastupto _t, xval(0(50)1000) by(posttran) keep(posttran _t surv2 lbsurv2 ubsurv2) norestore

. foreach Y of var surv2 ubsurv2 lbsurv2 {

. replace `Y'=1 if _t==0

. }

. list, abbr(32) sepby(posttran)

. eclplot surv2 lbsurv2 ubsurv2 _t, eplot(connected) rplot(rspike) estopts(connect(stairstep)) by(posttran)

. restore

Note that we reassign the missing survival probability,
and its missing confidence limits, at _t==0
to a sensible value of 1 for plotting purposes.

Author

Roger Newson, Imperial College London, UK.
Email: r.newson@imperial.ac.uk

Also see

Manual: [ST] sts generate

Online: help for sts generate

help for expgen, eclplot,
xcollapse, xcontract, descsave, parmest if installed