help for iedorep

Title

iedorep -- detects reproducibility errors in do-files automatically by running them multiple times and comparing the Stata state on a line-by-line basis between executions.

Syntax

iedorep "/path/do/do-file.do" , [ alldata allseed allsort ]

Description

iedorep provides a reproduciblity assessment of a do-file or project.
It should be used when a do-file or project is complete to detect any
instabilities or likely sources of replication errors.

iedorep will run the do-file twice and flag three types of possible
errors between the first and the second run of the do-file.
(1) Changes to data which leave the data at a different state
in the same line of the do-file. (Changes in the datasignature)
(2) Changes to the random number generator state
which result in a different state at the same line in the do-file. (Changes in the RNG state)
(3) Changes to the sort order of the data which leave the data
in a different sort order at the same line of the do-file. (Changes in the sortseed)

iedorep will first report a warning if the delimiter is changed in the do-file.
It is not designed to work with delimiters other than Stata's default (a newline).
It will also report a warning if you use clear all or ieboilstart;
specifically, it will re-write these commands to avoid calling discard,
as this command makes it impossible for iedorep to store the necessary data for its diagnostics.
If you experience unusual issues with iedorep, they are often memory issues;
in particular, the error "post posty not found" often indicates a memory management conflict.
Contact the maintainer to resolve these types of issues,
as they are typically hard-to-find bugs in memory management commands that are not often used.

iedorep is designed to be used targeting a master do-file (it will work on any file).
Its verbosity options allow the user to see all changes to data, RNG state,
and sort seed state, regardless of whether they are flagged as errors.

Options

The alldata, allseed , and allsort options will
report any time the data is changed, the RNG state changes, or the sort order of the data changed,
even if the result is consistent between the two do-file runs.
This is only recommended for advanced diagnostics;
for example, if reproducibility problems persist despite no errors appearing.
There are known cases where inconsistencies occur with very low
frequency and might not be detected in every given pair of runs.

Example verbose output

iedorep "$/run/iedorep/iedorep-target-1.do" ///
, alldata allsort allseed

+---------------------------------------------------------------+
| Line | Data | Seed | Sort | Subfile |
|------+----------------+-------------+---------------+---------|
| 5 | Changed | | | |
| 11 | Changed | | | |
| 13 | | | Sorted | |
| 15 | | | ERROR! Sorted | |
| 19 | ERROR! Changed | | | |
|------+----------------+-------------+---------------+---------|
| 20 | ERROR! Changed | ERROR! Used | | |
| 22 | | Used | | |
| 24 | ERROR! Changed | | ERROR! Sorted | |
| 26 | ERROR! Changed | | ERROR! Sorted | Yes |
+---------------------------------------------------------------+

Notes

In the example above, the rows without "ERROR!" flags would not appear
if the verbosity options were not specified.

We recommend resolving issues from the top down, since later issues
may be resolved by fixing earlier ones.
This is because many reproducibility issues will "cascade":
An inconsistent RNG state early in the do-file will cause many types of
later data creation (such as randomization assignment) to fail to reproduce.
Both issues will be fixed by resolving the first one.
Similarly, a single data inconsistency will cause every subsequent data check to fail
until new data is loaded that matches the data state in the first run,
but there may only be one true issue in the do-file.

Acknowledgements

...

Author

Main author: Benjamin Daniels, DIME Analytics, The World Bank Group

Please send bug-reports, suggestions and requests for clarifications
writing "ietoolkit iedorep" in the subject line to:

dimeanalytics@worldbank.org

You can also see the code, make comments to the code, see the version
history of the code, and submit additions or edits to the code through the GitHub repository of ietoolkit.