help for osgen (Roger Newson)

Add Python os_stat() file attributes to a xdir resultsset

osgen [if] [in] [ ,
filename(varname) dirname(varname) nodn

st_mode(newvarname) st_ino(newvarname) st_dev(newvarname) st_nlink(newvarname) st_uid(newvarname)

st_gid(newvarname) st_size(newvarname) st_atime(newvarname) st_mtime(newvarname) st_ctime(newvarname)

replace nodestring
]

Description

osgen is intended for use in output datasets (or resultssets)
produced by the xdir module of the xdir package,
which have one observation for each of a list of files,
and data on directory names and file names.
It inputs the file name variable and (optionally) the directory name variable,
and generates a list of new variables,
containing, in each observation, Python os_stat() attributes
of the corresponding file,
as recognised by the current operating system (OS).
These attributes include file sizes and creation, modification and access times.
The storage conventions for these OS statistics for files
may vary between operating environments,
but it might still be useful to be able to sort the extended xdir resultsset
by these attributes.
This package uses the Stata Python interface,
and requires the presence, in the user's operating environment,
of a version of Python compatible with the user's version of Stata.

Options

filename(varname) specifies an existing string variable,
containing the file names.
If absent, then it is set to filename.

dirname(varname) specifies an existing string variable,
containing the directory names.
If absent, then it is set to dirname.

nodn specifies that the directory name variable specified by dirname() will not be used.
This option can be useful,
if the file name variable specified by filename() contains the full file path for each file,
including the directory name.

st_mode(newvarname), st_ino(newvarname), st_dev(newvarname), st_nlink(newvarname), st_uid(newvarname),
st_gid(newvarname), st_size(newvarname), st_atime(newvarname), st_mtime(newvarname), and st_ctime(newvarname)
specify the names for the output variables to be created.
Each of these options specifies the name of a created variable containing the os_stat_result attribute of the same name
returned by the os_stat() command of the os module of Python.
In default, if one of these options is not specified,
then the new variable has the same name as the option.
These variables contain OS statistics for the corresponding files.

replace specifies that any existing variables with the same names as the generated output variables will be replaced.

nodestring specifies that the output variables will be created as string variables.
If nodestring is not specified, then the output variables will be converted to numeric,
preferably with the lowest storage type capable of storing the information.
The nodestring option may be useful,
if some of these numbers are extremely large and cannot be stored as numbers to exact precision,
as might happen if the numbers are numbers of bytes in multi-gigabyte files,
or times in nanoseconds since 1960.

Output variables created by osgen

The osgen package creates the following list of variables:

Default name Description

st_mode File mode (types and permissions)

st_ino File identification number (within device)

st_dev Device identifier

st_nlink Number of hard links

st_uid User ID of file owner

st_gid Group ID of file owner

st_size Size of file (bytes)

st_atime Last access time of file (seconds)

st_mtime Last modification time of file (seconds)

st_ctime Creation time of file (seconds)

The interpretation of these numbers may vary between operating environments,
which may have different conventions about precision and/or starting date-times.
However, it may be reasonable to assume that the sort order of file access, modification or creation dates
will be consistent between operating environments.
This should enable the user to create a sequence of output files under one operating environment
(such as a large data server using Linux),
and to sort them in the order in which these files were last modified under that environment,
and to send a list of these files, in creation order,
to another operating environment
(such as a user's own Windows machine).

Remarks

descgen is designed for use in output datasets (or resultssets),
with one observation for each of a list of files,
created using the xdir package,
which can be downloaded from SSC.
It inputs the variables whose default names are filename,
and (optionally) dirname,
containing the file names and directory names, respectively.
It outputs, in each observation,
the file attributes that may be output
by the Python command os_stat() of the os module.

Examples

Create xdir resultsset in current data frame:

. xdir, dir(.) pattern(*.*) fast

. describe, full

. list, abbr(32)

Generate new variables containing OS attributes using osgen:

. osgen, replace

. describe, full

. list, abbr(32)

Author

Roger Newson, Imperial College London, UK.
Email: r.newson@imperial.ac.uk

Acknowledgements

I would like to thank Rafal Raciborski of StataCorp for suggesting the use of the os module of Python
for accessing OS statistics about file sizes and modification dates in a way portable between operating environments,
and Zhao Xu of StataCorp for advising me on how to install a version of Python
compatible with my version of Stata.

Also see

Manual: [P] python, [D] dir

On-line: help for python, dir

help for xdir if installed