help txttool (SJX-X: dmXXXX)

Title

txttool -- Utilities for text analysis

Description

txttool provides a set of tools for managing and analyzing free-form text. The program integrates several built-in Stata functions with new text capabilities, including a utility to create a bag-of-words representation of text and an implementation of Porter’s word stemming algorithm.

Syntax

txttool varname [if] [in] [,
stem
stopwords(filename)
subwords(filename)
generate(newvarname)
replace
bagwords
prefix(string)
noclean
nooutput]

varname is the string variable containing the text to be processed.

Options

stem calls the Porter stemmer implementation to stem all of the words in varname.

stopwords(filename) indicates that the program should remove all instances of words contained in filename. The filename is simply a list of words in text file. Although a list of frequently used English words is supplied with txttool, users can use different lists of stopwords in different applications by specifying different filenames. Stopword lists without punctuation are recommended.

subwords(filename) indicates that the program should substitute instances of words in filename with another word in filename. The filename is a tab-delimited text file, where the first column is the word to be replaced and the second word is the text to substitute in. Users can use different lists of words to substitute in different applications by specifying different filenames. Subword lists without punctuation are recommended.

generate(newvarname) creates a new string variable newvarname containing the processed text of varname. The newvarname will be a copy of varname that has been stemmed, had the stopwords removed, words substituted, and/or cleaned, depending on the other options selected. Either generate or replace is required.

replace replaces the original text in varname with text that has been stemmed, had the stopwords removed, words substituted, and/or cleaned, depending on the other options selected. Either generate or replace is required.

bagwords tells txttool to create a bag of words representation of the text in varname. The bag of words representation consists of new variables, one for each unique word in varname, and populated with the count of the occurrences of each word. The new variables are named with the convention prefix_word, where prefix is optionally supplied by the user, and word is the unique word in the text. The options generate and bagwords can be used together to represent the processed text as a single column and counts of words.

prefix(string) supplies a prefix for the variables created in bagwords. If no prefix is supplied, the default is “w_”. Supplying a prefix will automatically invoke the bagwords option. Note that txttool does not know what variables will be created before processing the text, and so cannot confirm the absence of variables already named with the selected prefix. Errors will therefore result if the prefix chosen matches an existing variable.

noclean specifies that the program should not remove punctuation, extra white spaces, and special characters from varname. By default, txttool will clean and lowercase varname. The noclean option is not allowed with bagwords. In addition, because the Porter stemmer does not stem punctuation, and because the stopwords and subwords lists should not include punctuation, noclean should be used with caution.

nooutput suppresses the default output. By default, txttool reports the total number of words and the count of unique words before and after processing, as well as the time elapsed during processing. The nooutput option suppresses this output, which can save some time with large processing tasks.

Remarks

txttool options are process in the following order: cleaning, subwords, stopwords, stem, generate and bagwords.

By default, txttool will lowercase varname and clean the text by removing all characters except whitespace (ASCII code 32), numerals (ASCII codes 48-57) and letters (ASCII codes 97-122). To preserve any other characters, select the noclean. Note that the Porter stemmer stems only English words without punctuation, and may not function as expected with the noclean option. In addition, Stata does not allow any other characters to appear in variable names, so bagwords is not allowed with noclean.

Examples

. txtttol(exampletext), replace

. txtttol(exampletext), generate(newtext) noclean

. txtttol(exampletext), stem generate(newtext) subwords("C:\sublist.txt") stopwords("C:\stoplist.txt")

. txtttol(exampletext), stem generate(newtext) bagwords prefix("w_")

Author

Unislawa Williams, Spelman College

uwilliams@spelman.edu

Also see

Article: Stata Journal, volume XX, number XX: dm00XX

Help: [D] replace, [D] generate, [D] regexm()