help for iecompdup

Title

iecompdup -- Compares two duplicates and generate a list
of the variables where the duplicates are identical and a list of the variables where the
duplicates differ

For a more descriptive discussion on the intended usage and work flow of this
command please see the DIME Wiki.
Note that this command share wiki article with ieduplicates.

Syntax

iecompdup
id_varname
[if], id(id_value) [didifference keepdifference
keepother(varlist) more2ok]

options

id(id_value) value of the id_varname variable that is duplicated

didifference outputs the list with the variables for which the two observation differs. The default is to only store them in a local

keepdifference drops all but the variables for which the two observations differ

keepother(varlist) used together with keepdifference. Variables included in varlist are also kept

more2ok allows running the command on groups of more than two observations, although only the first two duplicates (in the order the data is sorted) are compared

Description

iecompdup compare all variables for observations that are duplicates in
the id_varname variable and the duplicated value is id(id_value). Duplicates can
be identified and corrected with its sister command ieduplicates. iecompdup
is intended to assist in the process of investigating why two observations are duplicated with respect
to id_varname, and what correction is appropriate.

iecompdup returns two locals r(matchvars) and r(diffvars). r(matchvars) returns
a list of the names of all variables for which the two observations
have identical values, unless both values are missing values or the empty
string. r(diffvars) returns a list of the names
of all variables where the two observations are not identical.

For example, if a duplicate is found in a dataset downloaded from a data
collection server (ODK or similar) and the duplicates were due to redundant submissions
of the same data, then r(diffvars) would only include the submission time
variable and any unique key used by the server. In such case, one observation can be
dropped without risking losing information, since it is an identical submission of
the exact same observation. (See Examples section below for a more detailed suggestion
on how to use the command. )

Options

id(id_value) is used to specify the ID value that the
duplicates share. Both text strings and numeric values are allowed.

didifference is used to display the list of all variables for which
the id_varname variable duplicates differ. The default is to provide this list in a local, and only
display the number of variables that differ.

keepdifference is used to return the data set with only the ID
variable and variables that differs between the duplicates. This means that the
command would drop all variables where the duplicates are identical or both
missing. It also drops all observations but the two duplicates compared.

keepother(varlist) is used to keep more variables than the variables
that differs between the duplicates when keepdifference is specified. The command can keep,
for example, a variable with information about who collected these data. This
option returns an error if it is specified not in conjunction with keepdifference.

more2ok allows running the command on groups of more than two observations,
although only the first two duplicates (in the order the data is sorted) are compared. In a group of three duplicates,
run the command three times on each combination of the three duplicates. A future update that
includes the possibility to compare more than one case is under consideration

Stored results

iecompdup stores the following results in r():

Locals

r(matchvars) a list of the variables where the duplicates has the same value

r(diffvars) a list of the variables where the duplicates has different values

Scalars

r(nummatch) The number of variables in r(matchvars)

r(numdiff) The number of variables in r(matchvars)

r(numnomiss) The number of variables for which at least one of
the duplicates has a non-missing value. By definition, r(numnomiss) equals
the sum of r(nummatch) and r(numdiff)

Examples

A series of examples on how to specify command, and how to evaluate output:

Example 1.

iecompdup HH_ID , id(55424) didifference

In the example above, let's say that there are two observations in the data set with the value 55424
for variable HH_ID. HH_ID holds an ID that was uniquely assigned to each household. Before continuing the analysis, one must
investigate why two observations were assigned the same ID. iecompdup is a great place to start.

Specifying the command as above compares the two observations that both
have a value of 55424 for variable id_varname. The output displayed will
only be number of non-missing variables for which the two observations have identical
values, and the number of non-missing variables for which the two observations
have different values. The list of those two sets of variables are stored as locals.
The data set is returned exactly as it was.

The locals stored in r(diffvars) and r(nummatch) can be used
to provide information on why the two observations are duplicates. A suggested
method to evaluate these two lists are presented in Example 2 below.

Example 2.

iecompdup HH_ID , id(55424) didifference

This example makes the same assumptions as example 1 that there are two
observations in the data set with the value 55424 for variable HH_ID. The only
difference is that the option didifference is specified. The output is the same
as example 1 but with the addition that the list stored in r(diffvars) is
displayed in the output window. The data set is returned exactly as it was.

The method to evaluate the output presented in this example focus on the
variables for which the duplicates are different. Therefore, start by looking at
the list of variables displayed by didifference. Do the variables with different values across the duplicates
contain observation data like "number of household members" or "annual income", or are they
submission information such as "submission ID", "server key" or "submission time"?
The answer to this question could suggest one of the three solutions below. Note
that this method should only be used as a guiding rule of thumb, all suggested
solutions should be evaluated qualitatively as well.

Solution 1. All variables contain submission information data. The
far most common mistake leading to duplicates in household surveys is that the same
observation data is submitted to the server twice. If that is the case, then only submission
information variables would be outputted by the command, not any observation data.
If this is the case, then you can safely delete either of the observations.

Solution 2. Most variables contain submission information data, but a few contain observation data. If a
few observation data variables are displayed together with submission information
variables then it is likely that it is the same observation but some variables
were edited after the first submission. Follow up with your field team to see
why some variables were changed. See the tips in example 3 below before following up.

Solution 3. Many variables contain observation data. If many
observation data variables are displayed together with submission data variables,
then it is likely that two different observations have accidentally been given the
same ID. That is especially likely if location variables or name variables are
different, or if the values for enumerator and/or supervisor are different. See the
tips in example 3 below before following up.

The cases listed above will solve the vast majority of duplicates encountered in
household surveys. The appropriate correction can afterwards be applied using the command ieduplicates.

Example 3.

iecompdup HH_ID , id(55424) didifference keepdifference keepother(village enumerator supervisor)

This example again makes the same assumptions as example 1 and example 2 that there are two
observations in the data set with the value 55424 for variable HH_ID. This
time keepdifference and keepother() are specified. Those two options
can be used to provide additional information to the field team when following up
based on solution 2 and solution 3 in example 2. keepdifference drops all
variables apart from id_varname and the variables in r(diffvars). Any
variables in keepother() are also kept. All observations apart from the
duplicates with the ID specified in id() are also dropped. This data can be
exported to excel and sent to a field team that can see how the observations differ.
In this example the field team can also see in which village the data was collected,
as well as the name of the enumerator and the supervisor. Any other information
helpful to the field team can be entered in keepother().

Example 4.

iecompdup HH_ID if inlist(key, "uuid:0003aad0", "uuid:0009baf1"), id(55424) didifference keepdifference keepother(village enumerator supervisor)

When there are several pairs or groups of duplicates, the command should be run
once for each pair or group, as iecompdup can oly compare two observations at a time.
In this case, use an if expression to select the observations to be compared.
Alternatively, you can use the more2ok option, which will compare the first two
duplicates observations.

Acknowledgements

I would like to acknowledge the help in testing and proofreading I received in relation to this command and help file from (in alphabetic order):

Michell Dong
Carlos Goes
Paula Gonzales

Author

All commands in ietoolkit is developed by DIME Analytics at DECIE, The World Bank's unit for Development Impact Evaluations.

Main author: Kristoffer Bjarkefur, DIME Analytics, The World Bank Group

Please send bug-reports, suggestions and requests for clarifications
writing "ietoolkit iecompdup" in the subject line to:

dimeanalytics@worldbank.org

You can also see the code, make comments to the code, see the version
history of the code, and submit additions or edits to the code through the GitHub repository of ietoolkit.