help for ieboilsave

Title

ieboilsave -- Checks that a dataset follows DECIE
standards for a data set, and tags the dataset with metadata.

For a more descriptive discussion on the intended usage and work flow of this
command please see the DIME Wiki.

Syntax

ieboilsave
, idvarname(varname) [dioutput missingok
tagnoname tagnohost]

options

idvarname(varname) specifies the ID variable
uniquely and fully identifying the data set

missingok regular missing values are allowed

dioutput display output summarizing results of tests made and meta data stored

tagnoname do not tag the data set with user name or the computer (host) name

tagnohost do not tag the data set with the computer (host) name

Description

ieboilsave standardizes the boilerplate (section of standardized
code) used at DECIE before saving a dataset. This includes checking that
the ID variable is uniquely and fully identifying the dataset. The test uses the
command isid, but provides a more useful output. Only one variable
is allowed to be the ID variable, see more in Notes on ID variables below.

The command also checks that no regular missing values are used. Missing values should
be replaced with the extended missing values .a, .b, ... , .z where each
extended missing value represents a reason for why the value is missing.

The command also tags meta data to the data set with information useful to
future users. The meta data is tagged to the data set using char.
Char stores meta data to the data set using an associative array,
see char for an explanation on how to access data stored with
char. The charnames (which is the equivalence to key or index in associative
arrays) is listed below. When applicable these values are taken from the
system parameters stored in c() (see creturn), and the
c() parameters used by this command are listed below. When a data
set already have these charnames, the old values are overwritten with the
new ones.

Charname Meta data associated with the charname

_dta[ie_idvar] stores the name of the ID variable that uniquely
and fully identifies the data set.

_dta[ie_version] stores the Stata version used (not installed,
see ieboilstart for more details) to create the data
set. Retrieved from c(version).

_dta[ie_date] stores the date the file was saved. Copying files,
sharing files over sync services or emails may change the time stamp shown
in folder. Retrieved from c(current_date).

_dta[ie_name] stores user name chosen when installing the
instance of Stata that was used when generating the file. Retrieved from
c(username). Storing this meta data is optional.

_dta[ie_host] stores computer name chosen when installing the
instance of the operative system that was used when generating the file.
Retrieved from c(hostname). Storing this meta data is optional.

_dta[ie_boilsave] stores a short summary of the result of
running ieboilsave. See option dioutput below for more details.

Options

idvarname(varname) specifies the ID variable that is
supposed to be fully and uniquely identifying the data set. This command
uses the command isid but provides a more helpful output in case
the ID variable has duplicates or missing values. Using multiple ID variables
to uniquely identify a data set is not best practice, and only one variable
is therefore allowed in varname. See Notes on ID variables below
read a justification for why it is bad practice.

dioutput displays the same information stored in _dta[ie_boilsave] in
the output window in Stata. This information includes the results of the ID
variable test, the missing values test and all the meta data stored
with char. Unless this option is specified, ieboilsave runs silently as
long as it does not cause any errors.

missingok allows the data set to have the standard missing values,
see missing values. Since changing regular missing values to extended
missing values is time consuming it might not always be a good use of a
Stata coder's time to do this for intermediary data sets. But since it should
be done for all final datasets, the default is to not allow regular missing
values.

tagnoname prevents the command from tagging the data set with metadata
containing user name and computer (host) name. Username and computer name
can be very useful when facing issues related to replicability. For privacy
reasons this can be disabled, but best practice is to keep it enabled at
least for all data sets that are not meant for public dissemination.

tagnohost is similar to tagnoname but it only prevents the
command from tagging the data set with metadata containing the computer name.
Specifying tagnohost is redundant if tagnoname is already
specified.

Examples

Example 1.

ieboilsave, idvarname(respondent_ID)

In the example above, the command checks that the variable respondent_ID
uniquely and fully identifies that data set, checks that there is no missing
values that are not among the extended missing values and saves meta data to the
data set using char.

Example 2.

ieboilsave, idvarname(respondent_ID) dioutput

The only difference between example 1 and this example is that in this
example the command outputs the information stored in _dta[ie_boilsave]. The
output will look similar to this:

. ieboilsave, idvarname(respondent_ID) dioutput

. ieboilsave ran successfully. The uniquely and fully identifying ID variable is hhid. This data set was created in Stata version 13.1, by user Kristoffer using computer Kristoffer-PC, on 27 Jan 2016. There are no regular missing values in this data set

Example 3.

ieboilsave, idvarname(respondent_ID)

local localname : char _dta[ie_boilsave]

di "`localname'"

Example 3 would generate exactly the same output as example 2 (formatted
slightly different) but this example shows how to display the information
in char _dta[ie_boilsave] at any point after running the command. For
example, if you receive new data set where _dta[ie_boilsave] is already
specified, then the two last lines of code is how you easiest access that
information in a readable way.

Notes on ID variables

The concept of Unique and Fully Identifying IDs (in short unique IDs) and
Unit of Observation are two concepts that cannot be emphasized enough in
data management best practices. The unit of observation is what each row
represents in a data set, and the unique ID should be unique for each
instance of the unit of observation. This is mostly the same unit as the
respondent during data collection.

For example, let's say the respondents during a data collection were farmers;
then the dataset is downloaded from the servers with farmers as the unit of observation.
However, let's say that the analysis was carried out at the plot level. The data
set prepared for the plot-level regressions no longer has farmer as
unit of observation, so it is plots and the dataset should be identified using
plot IDs not farmer IDs. If farmer IDs are unique for each farmer, and plot
IDs are unique among the plots for each farmer those two
IDs combined uniquely identify the data set. While it is technically true, it is
not good practice. Impact Evaluations run over many years and there is
likely going to be several different people working with the data set, and
the slightest confusion in ID variables can lead to large analysis
mistakes. It can lead to data sets merged incorrectly, that can lead to
duplicates and it can lead to several observations included multiple times
in a regression therefore inflating N and underestimating the p-value, causing
false positives.

Best practice is to always create a single variable that uniquely and fully
identifies every unit in the unit of observation before saving a data
set. Common practice is to make this the first variable in a data set
using order. It is also best practice to always start by making
sure you fully understand the unit of observation in datasets you get from
someone else. After you think you know the unit of observation, make sure
that you have a single variable that uniquely and fully identifies the
unit of observation in the data set.

These concepts are also central to modern database design. It is approached
somewhat differently as databases mostly consists of more than one dataset,
but the principles are the same. There are a lot of reading material online
search for primary keys and normalization in database design resources.

Acknowledgements

I would like to acknowledge the help in testing and proofreading I received in relation to this command and help file from (in alphabetic order):

Michell Dong
Paula Gonzales

Author

All commands in ietoolkit is developed by DIME Analytics at DECIE, The World Bank's unit for Development Impact Evaluations.

Main author: Kristoffer Bjarkefur, DIME Analytics, The World Bank Group

Please send bug-reports, suggestions and requests for clarifications
writing "ietoolkit ieboilsave" in the subject line to:

dimeanalytics@worldbank.org

You can also see the code, make comments to the code, see the version
history of the code, and submit additions or edits to the code through the GitHub repository of ietoolkit.