*! version 1.0.2 23Jan2019

Title

gduplicates -- Efficiently report, tag, or drop duplicate observations using C plugins.

Important: Please run gtools, upgrade to update gtools to
the latest stable version.

Syntax

This is a fast option to Stata's duplicates.
It is XX to XX times faster in Stata/IC and XX to XX times faster in MP

Report duplicates

gduplicates report [varlist] [if] [in]

List one example for each group of duplicates

gduplicates examples [varlist] [if] [in]
[, sorted options]

List all duplicates

gduplicates list [varlist] [if] [in]
[, sorted options]

Option sorted is required to fully mimic duplicates;
otherwise, gduplicates will not sort the list of examples or the
full list of duplicates. This default behavior improves performance but
may be harder to read.

Tag duplicates

gduplicates tag [varlist] [if] [in]
, generate(newvar)

Drop duplicates

gduplicates drop [if] [in]

gduplicates drop varlist [if] [in]
, force

Main
compress compress width of columns in both table and display formats

forcestrl Skip binary variable check and force gtools to read strL variables.

nocompress use display format of each variable

fast synonym for nocompress; no delay in output of large datasets

abbreviate(#) abbreviate variable names to # characters; default is ab(8)

string(#) truncate string variables to # characters; default is string(10)

Options
table force table format

display force display format

header display variable header once; default is table mode

noheader suppress variable header

header(#) display variable header every # lines

clean force table format with no divider or separator lines

divider draw divider lines between columns

separator(#) draw a separator line every # lines; default is separator(5)

sepby(varlist) draw a separator line whenever varlist values change

nolabel display numeric codes rather than label values

Summary
mean[(varlist)] add line reporting the mean for each of the (specified) variables

sum[(varlist)] add line reporting the sum for each of the (specified) variables

N[(varlist)] add line reporting the number of nonmissing values for each of the (specified) variables

labvar(varname) substitute Mean, Sum, or N for varname in last row of table

Advanced
constant[(varlist)] separate and list variables that are constant only once

notrim suppress string trimming

absolute display overall observation numbers when using by varlist:

nodotz display numerical values equal to .z as field of blanks

subvarname substitute characteristic for variable name in header

linesize(#) columns per line; default is linesize(79)

Description

gduplicates is a faster alternative to duplicates. It can
replicate every sub-command of duplicates; that is, it reports,
displays, lists, tags, or drops duplicate observations, depending on
the subcommand. Duplicates are observations with identical values
either on all variables if no varlist is specified or on a specified
varlist.

Note that for sub-commands examples and list the output is
NOT sorted by default. To mimic duplicates entirely, pass
option sorted when using those sub-commands.

gduplicates is part of the [R] gtools project.
In order to pass gtools options, use gtools(str).

Commands

gduplicates report produces a table showing observations
that occur as one or more copies and indicating how many observations are
"surplus" in the sense that they are the second (third, ...) copy of the first
of each group of duplicates.

gduplicates examples lists one example for each group of
duplicated observations. Each example represents the first occurrence of each
group in the dataset.

gduplicates list lists all duplicated observations.

gduplicates tag generates a variable representing the number of
duplicates for each observation. This will be 0 for all
unique observations.

gduplicates drop drops all but the first occurrence of each group
of duplicated observations. The word drop may not be abbreviated.

Any observations that do not satisfy specified if and/or in
conditions are ignored when you use report, examples, list,
or drop. The variable created by tag will have
missing values for such observations.

Options for duplicates examples and duplicates list

sorted Sort the output list. By default the list is left
unsorted to improve performance.

Main

compress, nocompress, fast, abbreviate(#),
string(#); see [D] list.

Options

table, display, header, noheader, header(#),
clean, divider, separator(#), sepby(varlist),
nolabel; see [D] list.

Summary

mean[(varlist)], sum[(varlist)],
N[(varlist)], labvar(varname); see
[D] list.

Advanced

constant[(varlist)], notrim, absolute, nodotz, subvarname, linesize(#); see [D] list.

Option for duplicates tag

generate(newvar) is required and specifies the name of a new variable
that will tag duplicates.

Option for duplicates drop

force specifies that observations duplicated with respect to a named
varlist be dropped. The force option is required when such
a varlist is given as a reminder that information may be lost by dropping
observations, given that those observations may differ on any variable
not included in varlist.

Examples

See duplicates or the
online documentation
for examples.

Author

Mauricio Caceres Bravo

mauricio.caceres.bravo@gmail.com

mcaceresb.github.io

Website

gduplicates is maintained as part of [R] gtools at github.com/mcaceresb/stata-gtools

Acknowledgment

This help file was based on StataCorp's own help file for duplicates

This project was largely inspired by Sergio Correia's ftools:
https://github.com/sergiocorreia/ftools.

The OSX version of gtools was implemented with invaluable help from @fbelotti;
see https://github.com/mcaceresb/stata-gtools/issues/11.

Also see

help for
gduplicates,
gisid,
gtools