*! version 1.4.3 28Apr2017

Title

scandata -- Scans the dataset and identifies variables fitting criteria of variable specifications (uppercase letters, length, mutated vowels, odd distributions) and partly corrects them.

Syntax

scandata [, options]

length(length_options) checks for variable names, value and variable labels having more than the amount of characters specified in length_options

nolabel checks for unlabeled values

oddvar(criteria) identifies variables with an odd distribution based on three criteria specified in criteria

umlaut checks for variable names, labels, value labels containing mutated vowels

strumlaut checks for strings containing mutated vowels

uppercase checks for variable names containing uppercase characters

all runs all checks above

correct corrects uppercase letters and/or mutated vowels

noprint shows a short summary of found variables instead of listing them in detail

Description

scandata scans the dataset and checks if variables fit certain criteria being important in light of data preparation. Those criteria can be chosen by using the options
length, nolabel, oddvar, umlaut and uppercase. At least one criterion must be named. To account for all criteria at once all can be used. Variables being found are presented in tables.
Furthermore, variables containing uppercase letters and/or mutated vowels can be corrected automatically by using correct. Thresholds used in length and oddvar can be changed by using further options (see option details).

Options

all accounts for every check available in the ado and can be combined with correct and noprint. If all is called only default values for length and oddvar are assigned.

correct can be used to correct variables found by scandata. Be aware that only variables regarding uppercase characters and mutated vowels can be corrected. For details see detailed description
of umlaut and uppercase.

length(length_options) checks if any variable name, value or variable label in the dataset contains more than a certain amount of characters. The exact amount can be specified by using the length_options
vallabel(#) (default is 119 (120 is the maximum value label length in SPSS)), varlabel(#) (default is 79) and varname(#) (default is 8). If no values are entered into the brackets the default values will be used.
In order to control just for one of those three possibilities the unnecessary options can be simply left out.

nolabel crawls through every value coded in the dataset checking if a label is assigned. This is useful, since little typos in programming of questionnaires can cause unlabled values. Furthermore,
it can be used to identify wild codes. To avoid unnecessary checks only those variables are scanned which contain an assigned value label. So, unlabeled values can only be found if other labels have been assigned to values already.

noprint can be used to create a short summary where only counts of variables being found are reported.

oddvar(criteria) checks if a variable may contain an odd distribution. An odd distribution is defined as fitting at least one of three criteria, incorporated in this program. Those criteria can be set on/off and modified
as well by using crit1(#), crit2(#) and crit3(#) within criteria. If no values are entered into the brackets the default values will be used. In order to control just for one or two of those three criteria the
unnecessary ones can be simply left out. Variables fitting at least one criterion are stored in r(odd_or] and those fitting all applied criteria are stored in r(odd_and).

The first criterion crit1(#) takes account of the amount of different values being in the variable. If the actual amount differs to a great extent from the available values named in the value label it could be a hint to
a misleading filter or program error. Thus, the variable is supposed to have an odd distribution. With crit1(#) the user can set the percentage threshold below which a variable is marked. (#) has to be a real number between zero
and one (the default is 0.01 and therefore 1 percent).

The second criterion crit2(#) looks for values having a distinct high frequency and therefore might hint at some type of program or filter error. Like for the first criterion the threshold being used can be modified,
but has to be a real number between zero and one. Here, scandata marks all variables which have certain value frequencies above the chosen threshold (the default is 0.95 and therefore 95 percent).

The third criterion crit3(#) looks for values having rather small observations hinting at some kind of wild codes or program/filter error. Therefore, (#) has to be an integer setting how many and less
observations are supposed to indicate an error in a variable. The default threshold is one.

strumlaut does the same as umlaut but only checks for string variables. Please note that string variables will be duplicated and get the suffix "_cmv" (corrected mutated vowels). Thus, the original strings are maintained.

umlaut goes through every variable name, label, value label and string and checks for existing mutated vowels. Concerning transformation in other formats or downwards compatibility to former Stata versions it can be useful
to replace them with their "normal" vowels. correct takes care of that step as well. Thus, "ä" becomes "ae", "ü" "ue", "ö" "oe" and "ß" "ss".
If those mutated vowels are upper-case characters they will be transformed in upper-case characters as long as the previous or following character are of an uppercase type, too. For example, "GRÜNE" becomes "GRUENE" but
"Übrige" becomes "Uebrige".

Make sure to translate the dataset into unicode if you use Stata 14 (see unicode). Otherwise, scandata cannot identify the mutated vowels. Furthermore, please note that quotation marks (',") in labels
cause problems in scanning the labels. Thus, they are transformed in the (hopefully) unique character and temporally. However, if your dataset contains such characters replace them with some other characters to avoid becoming
them quotation marks after scandata has finished.

uppercase accounts for checks regarding uppercase characters in the variable name. Upper-case characters, especially when mixed with lowercase characters, can be an issue when it comes to transformation in other formats.
If correct is named as well, all uppercase characters are turned into lowercase characters.

Returned results

Macros:

r(capvar) list of variables with uppercase letters in variable names

r(length_name) list of variables with more than the chosen threshold of characters in variable name

r(length_var) list of variables with more than the chosen threshold of characters in variable label

r(length_val) list of variables with more than the chosen threshold of characters in value label

r(umlaut_name) list of variables with mutated vowels in variable names

r(umlaut_var) list of variables with mutated vowels in variable labels

r(umlaut_val) list of numeric variables with mutated vowels in value labels

r(umlaut_str) list of string variables with mutated vowels

r(nolabel) list of numeric variables with unlabeled values

r(oddvar_or) list of variables which fit one of all applied criteria of an odd distribution

r(oddvar_and) list of variables which fit all applied criteria of an odd distribution

Examples

Using all available checks with default settings

. scandata, all

Using all available checks with default settings and correct uppercase characters and mutated vowels

. scandata, all correct

Checking for to many characters in labels and variable names, using default settings for variable labels and other thresholds for variable names and value labels

. scandata, length(varname(10) varlabel() vallabel(50))

Checking for mutated vowels in the dataset, no correction, display variables within a table

. scandata, umlaut

Checking for mutated vowels, correct them and supressing the table

. scandata, umlaut correct nop

Checking for an odd distribution with different values applying all criteria

. scandata, oddvar(crit1(0.02) crit2(0.99) crit3(3))

Checking for an odd distribution with different values applying only one criterion

. scandata, oddvar(crit2(0.99))

Using all checks with default settings but one different value for the third criterion

. scandata, all oddvar(crit1() crit2() crit3(2))

Author

Malte Kaukal, Hessen State Statistical Office malte.kaukal@gmx.de

Thank your for citing this software as follows:

Kaukal, Malte.2016. SCANDATA: Stata module to scan a dataset for specified characteristics. Available from
http://EconPapers.repec.org/RePEc:boc:bocode:s458186.

Acknowledgement

Regarding processing through each single value label this ado benefits a lot of the ado labellist written by Daniel Klein.

I would like to acknowledge the support of GESIS, which lead to the creation of the Stata command scandata.
Further I would like to thank my colleagues for their thoughtful comments and tests of earlier versions.