help for iecorrect

Title

iecorrect -- Modify data points in a dataset
using an external human-readable changelog (spreadsheet) and
maintain non-code documentation for all manual data point edits.

iecorrect provides a workflow to make changes (corrections)
to individual data points in a dataset without extensive Stata coding,
while documenting who made each change and why.
The changelog spreadsheet serves as both the instructions
for iecorrect to implement corrections as well as
a human-readable record of all changes made to data points.

iecorrect consists of two separate subcommands.
See the Intended workflow section
for a full description of how to use the command.
For a more descriptive discussion on the intended usage and workflow of this
command please see the DIME Wiki entry.

Functions

iecorrect template creates an Excel spreadsheet template
with empty columns for users to specify and document the corrections to be made by iecorrect apply.
This subcommand will be run once per data set to set up the changelog template,
then changes will be manually entered.

iecorrect apply reads an Excel changelog spreadsheet created
by filling out the template created by iecorrect template.
All data point corrections specified in the spreadsheet will be applied to the dataset in memory.
A do-file implementing the complete set of corrections may optionally be generated and saved.

Syntax

iecorrect template
using "/path/to/corrections/file.xlsx", idvar(varlist)

iecorrect apply
using "/path/to/corrections_file.xlsx", idvar(varlist)

[
sheets(string)
noisily
save(/path/to/do_file.do)
replace
break
]

Options

Corrections

idvar(varlist)
Specify the variable list that uniquely identifies the dataset.
Values from these variables will be used to select the observations to be changed.
This option is always required.

sheets(string)
Select which types of corrections will be made.
By default, corrections will be made from numeric, string, and drop sheets
if this option is not specified.
This option allows the user to select only a subset of them.

Documentation

save(string)
Save the do-file that makes modifications to the dataset
in the requested location.

replace
Overwrite the do-file that makes modifications to the dataset if it already exists.

noisily
Print the code and list of modifications as they run.

Error handling

break
Throw an error if inconsistencies are found in how the template was filled. By default,
if the variables listed in idvar(varlist) do not uniquely identify the data
or any lines filled in the template do not match any observation, only a warning message will be printed.

Intended workflow

When starting the data correction process for any dataset,
first run iecorrect template to create an empty Excel form.
This form should only be created once,
and then filled to indicate the changes that will be made to the data.
Once the form is filled, run iecorrect apply to apply the changes
indicated in the form to the dataset that is currently in memory.

The Excel report includes three sheets by default,
called string, numeric, and drop.
The other sheet may be added by specifying the other option.
Each of these sheets indicates one type of correction.
Each type of correction requires slightly different input from the user, discussed below.
Do not delete any of the sheets or the headers in the template form, or the command will not run properly.

Types of corrections

Corrections to string and numeric variables:

String and numeric variable corrections can be made through the string and numeric tabs of the Excel spreadsheet.

First, one column will appear for each specified idvar.
These must be filled with the value of the ID variable in the observation to be corrected.
If filled with *, corrections will be applied to all relevant observations
with the indicated current value of the variable to be corrected,
as well as matching values of any other completed idvar columns.

After the columns for the specified idvars, the sheet contains the following columns:

varname, which should be filled with the name of the string variable to be corrected.
Filling this column is required for this type of correction to run properly.

value, which should be filled with the desired corrected value
of the variable to be replaced in the selected observations.
This value will replace the current value once iecorrect apply is run.
Filling this column is required for this type of correction to run properly.

valuecurrent, which should be filled with the current, incorrect value
of the string variable in
the observation to be corrected. Filling this column is not required for this
type of correction to run properly, but either this column or the idvalue
column must be filled.

Dropping observations:

Corrections made by dropping observations are implemented through the
drop tab of the Excel spreadsheet. This tab has two required columns:

First, one column will appear for each specified idvar.
These must be filled with the value of the ID variable in the observation to be corrected.
If filled with *, all relevant observations will be dropped
with the indicated current value of the variable to be corrected,
as well as matching values of any other completed idvar columns.

n_obs, which must be filled with the exact number of observations
that should be dropped by the conditions specified in the corresponding line.
This column must always be filled for this type of correction to be performed
and it does not accept wildcards.
iecorrect will return an error if the number of observations
that have the ID values specified in the line does not match
the value entered in this column.
It will also return the number of observations that were detected matching the ID pattern,
so that you can check your expectations against the data.

Creating documentation

The columns initials and notes are included in all tabs created by iecorrect template,
and should be filled by the user to document the changes made to the code
when indicating which corrections should be made.

initials allows the team working with this data to keep track of who
decided on corrections.

notes allows the team working with this data to document how the
issue and the correct value were identified.

The columns date_last_changed and n_changes are included by iecorrect apply
when corrections are made to the data. This columns should not be manually edited by users.

date_last_changed documents the date when each correction was last
applied to the data. It is meant to allow users to identify any corrections that
have been included in the workbook, but not applied to the data.

n_changes documents the number of observations changed by each correction.
When a filled row in the filled workbook does make any actual changes to the data,
for example because it was filled incorrectly, this column will show 0 and
the command will return a warning.

Examples

Example 1: template subcommand

iecorrect template using "C:/myImpactEvaluation/baseline/documentation/Corrections.xlsx"

Specified like this, iecorrect will create a template Excel spreadsheet at
"C:/myImpactEvaluation/baseline/documentation/corrections.xlsx".
This template will be empty, and
each tab in this template must be filled by the user following the instructions above
before running the apply subcommand to implement the corrections.

Example 2: apply subcommand

iecorrect apply "C:/myImpactEvaluation/baseline/documentation/corrections.xlsx", idvar(key)

Specified like this, iecorrect starts by looking for the file
"C:/myImpactEvaluation/baseline/documentation/corrections.xlsx". If there is a report
with corrections, those corrections are applied to the data set.

Acknowledgements

We would like to acknowledge the help in testing and proofreading we received in relation to this command and help file from (in alphabetic order):

Benjamin Daniels
Timo Kapelari
Matteo Ruzzante
Ankriti Singh

Author

All commands in iefieldkit are developed by DIME Analytics at the World Bank's Development Impact Evaluations department.

Main authors: Luiza Cardoso de Andrade and Denisse Orozco Pereira, DIME Analytics, The World Bank Group

Please send bug-reports, suggestions and requests for clarifications
writing "iefieldkit iecorrect" in the subject line to:

dimeanalytics@worldbank.org

You can also see the code, make comments to the code, see the version
history of the code, and submit additions or edits to the code through the GitHub repository of iefieldkit.