help filelist

Title

filelist -- Recursively search directories for files

Syntax

filelist
,
[
directory(dirpath)
pattern(search_pattern)
save(stata_dataset)
replace
list
norecursive
maxdeep(#)
]

Description

filelist searches dirpath for files that match
search_pattern and continues searching recursively in all
its subdirectories. If directory(dirpath) is omitted,
the search starts from the current directory
(see pwd and cd).

By default, filelist will pick up all files, including
system files that are usually hidden. To target a specific
type of file using a pattern in the file name, use the
pattern(search_pattern) option.
The search_pattern string must conform to the rules of the strmatch()
function. For example, with pattern("*.csv"), filelist will return only
file names that end with ".csv".

filelist creates a dataset with 3 variables and as
many observations as there are matching files.
The dirname variable stores the file path to the file,
starting from the initial dirpath. The filename variable
stores the file name and fsize stores the file size in bytes.

You can use the save(stata_dataset) option to save the
results to disk instead of replacing the data in memory. Use
the replace option if stata_dataset already exists
and you want to overwrite it.

The list option is used to print the matched files
in Stata's Results window.

The maxdeep(#) can be used to control how many levels
deep filelist will search for files. maxdeep(1)
is equivalent to using the norecursive option and will
limit the search to the initial directory specified by dirpath.
The default is to search all subdirectories recursively.

Limitations

filelist can recursively scan a directory and return an unlimited
number of files (it will happily scan a whole hard disk if you ask for it).
Note however that filelist is written in Mata and unfortunately the
dir() function can only return 10,000 filenames from a single directory.
As of May 17, 2019, this hard coded limit is still present in the all versions
of Stata.

If the directory structure is deep enough,
the file path may exceed the maximum string length for variables of 244
characters in Stata version 9 to 12. When used with Stata 13
or higher, filelist can handle any length.

The Stata routines used by filelist
to discover the file size work with files
up to 2,147,483,647 bytes (2GB). For files
larger than 2GB, you need a 64-bit version
of Stata 13.1 (revision 03 Jun 2015) or higher.
Otherwise, fsize will
contain a missing value for that observation.

Examples

To find all files in the current directory and its subdirectories

. filelist

If there is a "main" directory within the current directory, you can
search for all Stata datasets in "main" using

. filelist, dir("main") pat("*.dta")

To search for all comma-separated data files in the "main" directory
within the current directory and save the results to disk

. filelist, dir("main") pat("*.csv") save("csv_datasets.dta")

You can run the following code if you want to use the saved search
results to append all csv data files

use "csv_datasets.dta", clear
local obs = _N
forvalues i=1/`obs' {
use "csv_datasets.dta" in `i', clear
local f = dirname + "/" + filename
insheet using "`f'", clear
gen source = "`f'"
tempfile save`i'
save "`save`i''"
}

use "`save1'", clear
forvalues i=2/`obs' {
append using "`save`i''"
}

Acknowledgments

A question on Statalist from Tim Evans was the stimulus for
writing this program.

Author

Robert Picard

picard@netbox.com

Also see

Stata: pwd, [D] cd, [D] dir, [P] macro -- Extended macro functions, [M-5] dir()

SSC: dirlist, dirtools, fs