help for dsconcat (Roger Newson)

Concatenate a list of Stata data files into the memory

dsconcat filename_list [ ,
append
subset([varlist] [if] [in])

dsid(newvarname) dsname(newvarname)
obsseq(newvarname)
nolabel nonotes noldsid
fast
]

where filename_list is a list of filenames separated by spaces.
If any filename in the list is specified without an extension, then
.dta is assumed.

Description

dsconcat is a multiple-file version of use or append.
It takes, as input, a list of filenames,
assumed to belong to Stata data files,
and creates a new dataset in memory,
containing a concatenation of the input data files,
which may either replace any existing dataset in memory
or be appended to any existing dataset in memory.
The new dataset contains all variables in all the input datasets (or a subset of variables
specified by the subset() option), and all observations in all the input datasets
(or a subset of observations specified by the subset() option),
ordered primarily by source dataset and secondarily by order of observations within each source dataset.
For any one variable in the output dataset in memory,
values of that variable are set to missing in any observation from an input dataset
not containing that variable.
Optionally, dsconcat creates new variables specifying,
for each observation, its input dataset of origin,
and/or the sequential order of the observation in its input dataset of origin.

Options

append specifies that the input datasets in files are to be appended to the existing dataset in memory,
if there is an existing dataset in memory.
If append is not specified, then any existing dataset in memory is overwritten.
Note that, if append is specified and there is no existing dataset in memory,
then the datasets in the input files are still concatenated into the memory,
as if append was not specified.

subset([varlist] [if] [in])
specifies a subset of variables and/or observations
in each of the input file datasets, to be included in the concatenated output dataset in memory.
The value of the subset() option is a combination of a varlist
and/or an if qualifier and/or an in qualifier.
Each of these is optional.
However, they must be valid for all input datasets,
according to the rules used by the use command.
Note that, if the append and subset() options are both specified,
then the subset() option only applies to observations and variables from the input file datasets,
and does not remove any observations or variables from the existing dataset in memory.
To do this, the user must use keep or drop before using dsconcat.

dsid(newvarname) specifies a new integer variable to be created,
containing, for each observation in the new dataset, the sequential order, in the
filename_list, of the input dataset of origin of the observation.
This sequential order is equal to 0 for observations from the original dataset in memory,
which are included only if the append option is specified,
and is equal to a positive integer for observations from the input file datasets.
If noldsid is not specified, then
dsconcat creates a value label for the newvarname with the same name,
assigning, to each positive integer i from 1 to the number of input filenames
in the filename_list, a label equal to the filename of the ith input dataset.
If a value label of that name already exists in one of the input datasets, and
nolabel is not specified, then dsconcat adds new labels,
but does not replace existing labels.

dsname(newvarname) specifies a new string variable containing,
for each observation in the new dataset, the name of the input dataset of origin
of that observation, truncated if necessary to the maximum string variable length
in the version of Stata being used.
This new string variable is equal to an empty string ("") for observations from the existing dataset in memory,
which are included only if the append option is specified.

obsseq(newvarname) specifies a new integer variable containing,
for each observation in the new dataset, the sequential order of that observation
in its input dataset of origin.
If the subset() option is specified, then
the sequential order of each observation from an input file dataset
is defined as its sequential order within the subset of observations in the original file dataset
specified by the subset() option,
excluding observations in the original file dataset excluded by the subset() option.

nolabel prevents dsconcat
from copying value label definitions from the input file datasets.
Note that, if the append option is also specified,
then the nolabel option does not affect value labels in the original dataset in memory.

nonotes prevents dsconcat
from copying notes from the input file datasets.
Note that, if the append option is also specified,
then the nonotes option does not affect notes in the original dataset in memory.

noldsid specifies that the new variable generated by the dsid() option will
have no value label.
This implies that the values of the new variable specified by the
dsid() option will be listed as dataset sequence numbers, not as dataset names.
This option is useful if the input datasets are very numerous
and/or are repeated and/or are temporary files with uninformative names.
It is ignored if no dsid() option is specified.

fast is an option for programmers.
It specifies that any existing dataset in the memory will not necessarily be restored,
if dsconcat fails or the user presses Break.
If fast is not specified, then the existing dataset is restored
if dsconcat fails or the user presses Break,
and this precaution requires a certain amount of time-consuming file processing.
Note that the fast option is ignored
if the append option and the subset() option are both specified,
because then a certain amount of file processing is necessary,
even if dsconcat terminates without error or interruption.

Remarks

dsconcat is a multi-file version of use or append.
However, dsconcat is different from use in that, unless the append option is specified,
dsconcat overwrites existing datasets in memory automatically (as collapse and contract do),
instead of requiring a clear option (as use does).

dsconcat does not sort the data, or check uniqueness of observations in the concatenated dataset.
Therefore, users are advised to use the keyby package, downloadable from SSC,
if they wish to enforce the relational database model,
and create datasets whose observations are sorted and identified by a list of variables known as a primary key.
In the relational database model, a dataset is viewed as a mathematical function,
whose domain is the set of available combinations of values of the primary key variables,
and whose range is the set of all possible combinations of values
for the variables not included in the primary key.
See Examples below for an example using the keybygen module of the keyby package.

Examples

. dsconcat auto1 auto2 auto3 auto4, dsid(dsseq) obs(obsnum)

. sort dsseq obsnum

. dsconcat "Microsoft is inferior" Unix_is_superior IdontknowaboutMacOS

. dsconcat auto1 auto2 auto3 auto4, append dsid(dsseq) obs(obsseq)

The following example creates a dataset containing variables make, foreign, mpg
and weight in the first 53 observations of each of the datasets auto1, auto2,
auto3 and auto4, with the input dataset name stored in the new string variable dslab:

. dsconcat auto1 auto2 auto3 auto4, subset(make foreign mpg weight in 1/53) dsn(dslab)

The following example uses the saved result r(sortedby1) (see Saved results below),
saved by dsconcat,
together with the keybygen module of the keyby package,
which can be downloaded from SSC.
The concatenated dataset eventually created in memory has observations
sorted primarily in the order of the input dataset from which the observation was input
(specified by the new variable dsseq),
secondarily by the variable (or variables) defining the sort order of the first input dataset auto1,
and thirdly by the order of the observation within its by-group in the input dataset
(specified by the new variable obsseq).
The keybygen command ensures that the observations in the dataset are identified uniquely
by the values of the variables defining the sort order.

. dsconcat auto1 auto2 auto3 auto4, dsid(dsseq) noldsid

. keybygen dsseq `r(sortedby)', gene(obsseq)

. describe

Saved results

dsconcat saves the following in r():

Scalars

r(nobs) total number of observations in concatenated dataset

r(ndset) number of input dataset files

Macros

r(sortedby) list of variables specifying sort order of first input dataset

The result r(sortedby) contains the sort order of the dataset in memory,
if the append option is specified.
or the sort order of the dataset in the first input file,
if the append option is not specified.
Note that dsconcat does not perform any sorting of the data.
It is the user's responsibility to do this,
possibly using the result r(sortedby).

Author

Roger Newson, Imperial College London, UK.
Email: r.newson@imperial.ac.uk

Also see

Manual: [D] append, [D] drop, [D] save, [D] use

On-line: append, keep, drop, save, use

keyby, keybygen if installed