*! version 1.1 05apr2017

Title

strkeep -- Clean string variables by keeping only whitelisted ASCII characters

Syntax

strkeep
varlist
[if] [in] , options

strkeep options

Generate Options (must specify exactly one)
generate(varname|stub) generate and store results in varname or in stub1, stub2, etc.

replace replace varlist with cleaned results

Whitelist Options (must specify at least one)
alpha keep all letters of the alphabet, both lowercase and uppercase

lowercase keep lowercase letters of the alphabet

uppercase keep uppercase letters of the alphabet

numeric keep Arabic numerals

keep(string) keep characters in string, which is interpreted as individual characters

strlower()/strupper() Options (optional)
strlower results returned in lowercase format

strupper results returned in uppercase format

Substitution Option (optional)
sub(string) replace any deleted character with string, which must be 1 character in length;
any character used in sub() will be automatically added to the whitelist

Description

strkeep cleans a string variable or varlist by keeping only certain ASCII characters.
In essence, this command is a "whitelist" command, while a function like
subinstr() is a "blacklist" command. Instead of finding characters that you want
to remove, strkeep finds characters that you want to keep and removes all other characters.

When using strkeep, you must specify one and only one of the
generate options and at least one of the
whitelist options. The substitution option
is optional.

At least one whitelist option must be specified, but more than one can be specified at a time.
If multiple whitelist options are specified, then all of the corresponding characters are
whitelisted and are kept in the strings. Users can define specific characters to be kept by
using the keep() option. N.B., keep("a b") will keep the characters "a",
"b", and " ", as a single space is a character.

The strlower/strupper options will perform the corresponding functions on the final
results of strkeep. The options are mutually exclusive and are performed at the
end of the function. If the option uppercase is specified along with strlower, then only
uppercase letters will be kept, but the result will show those uppercase letters
converted to lowercase letters. For example, in "ABcdEFG", "ABEFG" will be kept,
but the resulting string will be "abefg" as strkeep will then perform strlower()
on the results.

The substitution option is completely optional, and of limited use. If sub() is not specified,
strkeep simply deletes any character that is not whitelisted. sub() can
only accept one single character and will reject strings with multiple characters.
Keep in mind that any character used as the substitution character will be added to
the whitelist.

If a varlist is passed to strkeep and generate(stub) is specified, the command
will create multiple new variables. Specifically, the command will create new variables
named stub1, stub2, etc., corresponding to the number and order of the variables
in the varlist. stub has a character limit below 32 characters (the variable name
limit in Stata) and dependent on the total number of variables in varlist.
If 100 variables are passed in varlist, then stub can only be 29 characters long, to
allow for stub100 as a variable name.

Examples

Setup

. sysuse auto.dta

Clean the string variable make to keep only lowercase and uppercase letters,
and generate a new variable

. strkeep make, generate(make_alpha) alpha

The above is identical to the below

. strkeep make, generate(make_alpha) lowercase uppercase

The above is identical to the below

. strkeep make, generate(make_alpha) keep("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")

Setup

. sysuse auto.dta

Clean the string variable make to keep only lowercase letters and numbers

. strkeep make, generate(make_clean) lowercase numeric

Setup

. sysuse auto.dta

Clean the string variable make to keep all letters, numbers, and underscores, and
replace everything else with an underscore

. strkeep make, generate(make_clean) alpha numeric sub("_")

Acknowledgements

Thank you to Lenny Wainstein for help in checking the code of strkeep.

Author

Roger Chu, Research for Action, rchu@researchforaction.org

Version History

1.1 - April 5, 2017 - added strlower and strupper options

1.0 - November 3, 2016