*! version 2.0.1 30jul2026 Eric A. Booth and Elizabeth Teas

help for combineallv2.0.1

Title

combineall -- Combine (append, merge, or joinby) or convert every file in a directory, with an optional vintage-aware harmonization layer

Syntax

combineall [using filename] [, options]

filename names the combined output dataset (.dta may be given or
omitted). If using is omitted, output goes to
combineall_output.dta inside directory(). Every option is
optional, so a bare combineall is legal: it converts every
.csv file in the working directory to .dta.

engine options (2011)

core
cmethod(method) combine method: append, merge, joinby, or convertonly (the default)

directory(path) folder containing the files to combine or convert; default is the working directory

replace overwrite the using output file if it exists

keepconverted keep the per-file converted .dta copies; implied by cmethod(convertonly)

combine (append/merge/joinby)
mtype(type) merge type: 1:1 (default), m:m, 1:m, or m:1

mvars(varlist) key variables for merge or joinby; required with those methods

_merge create one match-status variable per combined file, named _filestem

other options options legal for append, merge, or joinby pass through

file handling
filetype(ext) extension of the input files: csv (default), dta, txt, xlsx, xls, xml, or any text extension

fileid(newvar) create newvar holding each observation's source filename

delimiter(char) comma (default), tab, or a delimiter character such as ";"

prefix(string) string added to the beginning of each converted filename

suffix(string) string added to the end of each converted filename

tostring tostring every variable during conversion, using each variable's display format (lossy for numerics)

xmlopts(options) options passed to xmluse when filetype(xml)

harmonization options (v2.0.0; cmethod(append) only)

map(filename) CSV rename map with columns oldname,newname,firstyear,lastyear; activates the layer

year(spec) how to extract the year from each filename: a starting position or a regular expression; default is the first 4-digit run 19xx/20xx

strict error (rather than report) when a mapped oldname is absent from a file its window covers

Description

combineall sweeps a directory and combines every file of one type into
a single dataset. Text files (csv, txt, and any other delimited
extension), Excel files (xlsx/xls), and XML files are first
converted to .dta; native .dta files are used directly. The
files are then combined by append, merge, or joinby,
or simply converted and left in place with cmethod(convertonly).
The engine dates to 2011 (v1.0.0) and its option set is unchanged.

v2.0.0 adds a harmonization layer for stacking yearly file releases, active
under cmethod(append). Agencies publish one file per year, and
element names mutate across vintages: a column called one thing in 2015 is
renamed by 2020, so a naive append stacks unrelated variables. With
map(), combineall extracts the 4-digit year stamped in each
filename, sorts the files by year, applies only the renames whose
[firstyear,lastyear] window covers that year, generates int year,
and appends everything into one harmonized panel.

Under map(), every rename is written down twice. The map CSV is the
human-readable record of the harmonization decisions, and each renamed
variable carries a characteristic,
char varname[source], recording the original name, source
file, and year, so the panel documents its own provenance. After stacking,
combineall prints a harmonization table (each final variable by the
years in which it has data) and lists any file where a mapped oldname
covered by the window was absent. That absence is a report, not an error,
unless you add strict.

Engine options

core options

cmethod(method) sets the combine method: append, merge,
joinby, or convertonly. The default, convertonly,
converts each file to .dta in directory() and combines nothing.

directory(path) names the folder containing the input files. The
default is the current working directory. The output using file is
excluded from the input list by name, so an output saved inside
directory() is not re-consumed on a second run.

replace overwrites the using output file. Without it,
combineall refuses to overwrite an existing output (error 198).

keepconverted keeps the intermediate converted .dta copies
(named prefix'filestem'suffix.dta) after
combining. Without it they are deleted once combined.
cmethod(convertonly) implies keepconverted; in v1.0.0 the two
had to be specified together.

combine (append/merge/joinby) options

mtype(type) sets the merge type (1:1, m:m,
1:m, or m:1). Default 1:1.

mvars(varlist) names the key variables for merge or
joinby; required with those methods. The keys are created as empty
string variables in the seed master, so they must be strings in every input
file; see Remarks and limits.

_merge creates one match-status variable per combined file, named
_filestem (via gen() for merge, _merge() for
joinby). File stems must therefore be valid variable-name material.

Any other option legal for append, merge, or joinby
passes through, for example unmatched(both) for joinby.

file-handling options

filetype(ext) sets which files are swept up, by extension.
csv (default), txt, raw, out, log, and any
unrecognized extension are read with import delimited;
xlsx/xls with import excel (firstrow);
xml with xmluse; dta with use. (v1.0.0 used
insheet and read xls as text; v2.0.0 reads Excel properly.)

fileid(newvar) creates newvar containing each observation's
source filename, for example enr_2019.csv.

delimiter(char) sets the text delimiter: comma (default),
tab, or a character such as ";".

prefix(string) and suffix(string) decorate the converted
filenames, which is how you keep converted copies from colliding with
existing files. Periods are removed from both.

tostring converts every variable to string during conversion, for
files whose types wobble across sources. The conversion is lossy for
numeric variables. Each variable is passed to
tostring varname, force replace usedisplayformat,
and usedisplayformat is the catch: a value is written as its display
format renders it, not at full numeric precision. A double holding 1/3
under the default %10.0g format becomes the string .33333333,
and destring cannot recover the digits that were dropped.
force allows that lossy conversion to proceed, and combineall
runs the conversion quietly, so tostring's own loss-of-information
message is never displayed. Reserve tostring for identifier-like
columns whose storage type wobbles across files, not for measured
quantities you intend to analyze at full precision.

xmlopts(options) passes options to xmluse, for example
xmlopts(doctype(excel) firstrow).

Harmonization options (v2.0.0)

These options require cmethod(append); combining them with any other
method is an error (198). year() and strict additionally
require map().

map(filename) names the rename map, a plain CSV described under
The map file below. Specifying map()
activates the whole layer: year extraction, year-order sorting,
vintage-window renames, the year variable, [source] provenance
characteristics, and the harmonization table. A map containing only the
header row stamps year without renaming anything.

year(spec) controls year extraction from each filename (extension
excluded). By default the first 4-digit run beginning 19 or 20 is used, so
tapr_2019_campus.csv yields 2019. Two overrides are available:

a number is a 1-based starting position; year(14) reads 4
characters starting at position 14 of the filename.

anything else is treated as a regular expression
(ICU/ustrregexm syntax). If it contains a capture group, group 1 is
the year; otherwise the whole match is. Example:
year("run_([0-9][0-9][0-9][0-9])").

strict turns the absent-oldname report into an error (111). Use
it when the map is a contract: every file whose year is covered by a window
must contain that oldname.

The map file

The map is a CSV with a header row and four columns:

oldname,newname,firstyear,lastyear

enroll_cnt,enrollment,,2020

mscore,mathscore,2019,2019

A row means: in files whose year falls inside [firstyear,lastyear],
rename oldname to newname. A blank firstyear or
lastyear leaves that end open, so the first row above reads: through
2020, enroll_cnt is what enrollment was called. Rows apply in
file order; once an oldname is renamed in a file, later rows naming
the same oldname are skipped for that file. A row whose oldname
equals its newname simply asserts the variable's presence (useful with
strict).

Matching is exact and case-sensitive, and text files are imported with case
preserved; write the map in the case the source files use. Keep the map
file outside directory() (or use a different extension than
filetype()), or it will be swept up as an input file.

Stored results

combineall stores in r():

r(n_files) number of files converted/combined

r(output) path of the combined file (not convertonly)

r(n_vars) number of variables in the panel, including year (map() only)

r(n_missing) number of expected-but-absent oldname reports (map() only)

r(years) distinct years stacked, ascending (map() only)

Examples

Each block below is self-contained. It builds its own input files from
sysuse auto, creates the folders it writes into, and runs as
shown in an empty working directory. Stata does not create directories on
demand, so a mkdir precedes every block that writes into a
subdirectory; capture is used so a block also survives a second run,
when the folder already exists.

Each block builds its inputs in a folder of its own on purpose. Under
filetype(dta) with no prefix() or suffix() the "converted"
copy is the source file itself, so a block that adds fileid() writes
that column back into the files it read, and a later command pointed at the
same folder would silently inherit it; see
Remarks and limits.

Convert every CSV in a folder to .dta (converted copies are
named zname.dta):

. capture mkdir "raw"

. sysuse auto, clear

. keep make price mpg

. export delimited using "raw/cars_2019.csv", replace

. export delimited using "raw/cars_2020.csv", replace

. combineall, cmethod(convertonly) directory("raw/") prefix(z)

Append every .dta file in a folder, tagging each row with its
source file. Because filetype(dta) rewrites the sources in place,
this block leaves srcfile behind in pieces/part1.dta and
pieces/part2.dta:

. capture mkdir "pieces"

. capture mkdir "built"

. sysuse auto, clear

. keep make price mpg

. save "pieces/part1.dta", replace

. save "pieces/part2.dta", replace

. combineall using "built/all.dta", cmethod(append) directory("pieces/") filetype(dta) fileid(srcfile) replace

Merge every file on a string key, with one match-status variable
per file. The key must be a string in every input file. This block builds
a fresh fixture in keys/ rather than reusing pieces/, which the
block above rewrote when it added srcfile:

. capture mkdir "keys"

. capture mkdir "built"

. sysuse auto, clear

. gen str3 campusid = string(_n, "%03.0f")

. keep campusid price

. save "keys/prices.dta", replace

. sysuse auto, clear

. gen str3 campusid = string(_n, "%03.0f")

. keep campusid mpg

. save "keys/mileage.dta", replace

. combineall using "built/wide.dta", cmethod(merge) directory("keys/") filetype(dta) mvars(campusid) _merge replace

Stack yearly vintages with a rename map. Build two tiny yearly
files whose price column changes name, write a two-line map, and stack.
The 2019 file calls the column price; the 2020 vintage renamed it
price_usd, and the map folds it back:

. capture mkdir "raw"

. capture mkdir "built"

. sysuse auto, clear

. keep make price mpg

. export delimited using "raw/cars_2019.csv", replace

. rename price price_usd

. export delimited using "raw/cars_2020.csv", replace

. file open m using "map.csv", write replace

. file write m "oldname,newname,firstyear,lastyear" _n

. file write m "price_usd,price,2020," _n

. file close m

. combineall using "built/cars_panel", cmethod(append) directory("raw/") map("map.csv") replace

. use "built/cars_panel.dta", clear

. char list price[source]

. tabulate year

Filenames where the year is not the first 4-digit run. The
default extractor would read 2020 out of batch2020run_2019.csv, which
is the batch identifier rather than the year. A starting position and an
explicit regular expression both pick out 2019 instead:

. capture mkdir "pos"

. capture mkdir "built"

. sysuse auto, clear

. keep make price mpg

. export delimited using "pos/batch2020run_2019.csv", replace

. file open m using "posmap.csv", write replace

. file write m "oldname,newname,firstyear,lastyear" _n

. file write m "price,price_usd,2019,2019" _n

. file close m

. combineall using "built/p", cmethod(append) directory("pos/") map("posmap.csv") year(14) replace

. combineall using "built/p", cmethod(append) directory("pos/") map("posmap.csv") year("run_([0-9][0-9][0-9][0-9])") replace

Insist the map is fully satisfied. A map row whose oldname
equals its newname asserts that the variable is present. With
strict, a file whose year the window covers but which lacks that
variable stops the run with error 111 instead of producing a report:

. capture mkdir "contract"

. capture mkdir "built"

. sysuse auto, clear

. keep make price mpg

. export delimited using "contract/cars_2021.csv", replace

. file open cm using "contractmap.csv", write replace

. file write cm "oldname,newname,firstyear,lastyear" _n

. file write cm "price,price,," _n

. file close cm

. combineall using "built/contract", cmethod(append) directory("contract/") map("contractmap.csv") strict replace

Remarks and limits

Output goes to the using file; memory is untouched. Unlike commands
that leave results in memory, combineall preserves and restores your
current data and writes the combined dataset to disk. Load it with
use afterward.

Appends use force. The 2011 engine appends with force, so a
variable that is string in one file and numeric in another is coerced (the
offending values become missing) rather than stopping with an error. This
engine behavior is kept in v2.0.0, including under map(). Check the
harmonization table for unexpected gaps. Importing everything as strings
with tostring and then applying destring deliberately is one
way around the coercion, but that round trip is itself lossy.
tostring is run with usedisplayformat, so a numeric value is
written as its display format renders it and the digits beyond it are gone
before destring ever sees the column (see
tostring above). Reserve that route for
identifier-like columns.

Merge and joinby keys must be strings. The engine seeds an empty
master whose key variables are created as empty strings; a numeric key in
the input files stops the first merge with a type mismatch (r(106)).
Convert keys to string in the sources, or combine with
cmethod(append) and reshape afterward.

joinby needs unmatched(both). Because the seed master matches
nothing, a plain joinby drops everything. Pass
unmatched(both) (as the 2011 examples did) and drop the seed
artifacts you do not want.

filetype(dta) can rewrite sources in place. With filetype(dta)
and empty prefix()/suffix(), the "converted" copy is the source
file itself, so fileid(), tostring, or map() renames are
written back into the source files. Specify prefix() or
suffix() to keep the originals pristine.

One year per file. The harmonization layer stamps one year per file,
taken from the filename; files containing multiple years should be split
upstream. A file that already contains a variable named year stops
with an error and a hint to move it aside with a map row.

If strict (or any error) fires mid-run, the partially combined
using file may remain on disk; rerun with replace after fixing
the problem.

Not yet handled: recursing into subdirectories, and filenames whose
stems are not valid tokens for the _merge variable names.

Authors

Eric A. Booth, Sr Researcher, Texas 2036

eric.a.booth@gmail.com

Elizabeth Teas, Sr Research Scientist, Far Harbor, LLC

elizabeth@farharbor.com

combineall v1.0.0 was first released in April 2011; v2.0.0 (2026) folds in
the vintage-harmonization layer developed for the authors'
applied-evaluation book. MIT-licensed.

Also see

On-line: help for append, merge, joinby,
import delimited, import excel, xmluse, char,
use