help textfind

Title

textfind -- identify, analyze, and convert text entries into
categorical data

Syntax

textfind
varlist
[if] [in]
[,
keyword("string1" "string2" ...)
but("string1" "string2" ...)
nocase
exact
or
notable
tag(newvar)
nfinds
length
position
tfidf]

Description

textfind is a data-driven program that identifies, analyzes, and converts
textual data into categorical variables for further use in quantitative
analysis. It uses regular expressions to find one (or more) keyword and
exclusion (i.e. n-grams), reporting six statistics summarizing search
quality: the number of observations in the dataset that were matched; the number
of word occurrences per observation; the textual length in which word is found;
the position at which the word was first found; the term frequency-inverse
document frequency (tf-idf) of the word used in the search; and the p-value of a
means comparison test between samples identified by different search criteria.

Options

keyword("string1" "string2"
...) is the main search option. It looks up "string1",
"string2", ..., in each observation of varlist, where string can be
text, numbers, or any other ustrregexm() search criteria.

but("string1" "string2" ...
) is the main exclusion option. It looks up "string1", "string2",
..., in each observation of varlist, where string can be text,
numbers, or any other ustrregexm() search criteria, and removes matches
found with keyword().

nocase performs a case-insensitive search.

exact performs an exact search of keyword() in varlist and
only matches observations that are entirely equal to "string1",
"string2", ..., etc.

or performs an alternative match for multiple entries in
keyword(). The default is an additive search of "string1" and
"string2" ...

notable asks Stata not to return the table of summary statistics.

tag(newvar) generates one variable called newvar marking all
observations that were found under criteria keyword() and but().

nfinds generates one variable per "string" in keyword()
containing the number of occurrences of "string" in each observation.
Default variable names are myvar1_nfinds, myvar2_nfinds,
..., for "string1", "string2", ..., etc.

length generates new variable myvar_length containing
the word length of each variable in varlist for which search criteria is
found.

position generates one variable per "string" in
keyword() containing the position where "string" was first found in
each observation. Default variable names are myvar1_pos,
myvar2_pos, ..., for string1, string2, ..., etc.

tfidf generates one variable per "string" in keyword()
containing the term frequency-inverse document frequency statistic of
"text" in each observation. Default variable names are
myvar1_tfidf, myvar2_tfidf, ..., for string1,
string2, ..., etc.

Remarks

textfind increases Stata's capabilities for conducting content analysis.
Beyond standard keyword search made possible by string functions,
textfind allows users to use multiple keyword and exclusion criteria to
identify observations in the dataset.

In particular, textfind has three important features: (i) it makes use of
regular expressions for highly-complex search patterns; (ii) it produces six
measures of textual match quality, including a means comparison test across
search criteria; (iii) it uses Unicode encoding, instead of ASCII, thus making
it compatible with non-English text excerpts and strings.

The program produces a summary table with six statistics by each keyword and
exclusion.

(1) Total Finds (exclusions): returns the number of observations
found by search criteria in keyword() or but(). If both criteria
have been specified, but() removes finds identified by keyword().

(2) Average Finds (exclusions): returns the average number of
occurrences of strings in keyword() [or exclusions from but()]
by observation.

(3) Average Length: returns the average length (in words) of text
in observations where keyword() [or but()] were [not] found.

(4) Average Position: returns the average position in which
keyword() or but() were found.

(5) Average TF-IDF: returns the average tf-idf statistic for all
observations where keyword() or but() were found.

(6) Means test: returns the p-value of a t-test on the difference
of means across two immediate samples. It measures the improvement of using
n vs. n-1 search criteria when identifying a subsample of textual
observations.

Examples

. use https://github.com/aassumpcao/textfind/raw/master/CivilServantsNeverland.dta

This is a hypothetical dataset reporting positions of 5,000 government officials
in Neverland. We want to identify all observations which contain the unigram
"officer" but which do not have the unigram "level". The usual steps would be:
(1) find observations using keyword "officer";
(2) find observations not containing keyword "level";
(3) find observations with keyword "officer" but remove observations which also
contain keyword "level".

. tab post if ustrregexm(post, "officer", 1) == 1

post | Freq. Percent Cum.
-----------------------------+-----------------------------------
Senior Hook Security Officer | 525 34.79 34.79
fairy officer (senior level) | 480 31.81 66.60
officer | 504 33.40 100.00
-----------------------------+-----------------------------------
Total | 1,509 100.00

. tab post if ustrregexm(post, "level", 1) == 0

post | Freq. Percent Cum.
-----------------------------+-----------------------------------
Analyst | 527 11.66 11.66
Senior Hook Security Officer | 525 11.62 23.27
analist | 501 11.08 34.36
analyst | 476 10.53 44.89
fairy analyst | 512 11.33 56.22
officer | 504 11.15 67.37
piracy analyst | 492 10.88 78.25
senior manager | 507 11.22 89.47
senior piracy analyst | 476 10.53 100.00
-----------------------------+-----------------------------------
Total | 4,520 100.00

. tab post if ustrregexm(post, "officer", 1) == 1 & ustrregexm(post, "level", 1) == 0

post | Freq. Percent Cum.
-----------------------------+-----------------------------------
Senior Hook Security Officer | 525 51.02 51.02
officer | 504 48.98 100.00
-----------------------------+-----------------------------------
Total | 1,029 100.00

Here is the result using textfind. It identifies the same observations as
the commands above but it does so in one line of code and it returns six
statistics on the quality of match.

. textfind post, key("officer") but("level") nocase

Summary Table
--------------------------------------------------------------------------------
variable: post
n: 5000 Average Means
Total ----------------------------------------- test
keyword(s) Finds Finds Length Position TF-IDF p-value
--------------------------------------------------------------------------------
officer 1509 1 3.63419 2.36183 .567835 8.e-188
--------------------------------------------------------------------------------
Total 1029 1 2.53061 2.53061 .975933 0
--------------------------------------------------------------------------------
exclusion(s):
"level"

Stored Results

textfind stores the following in r():

Scalars

r(fvarmn) word m = [1,2,...], statistic n = [1,6],
found in each var from varlist.

r(nvarmn) word m = [1,2,...], statistic n = [1,6], not
found in each var from varlist.

r(max) maximum number of words in largest string var in
varlist.

r(nkey) number of find criteria.

r(mbut) number of exclusion criteria.

Macros

r(allkey) all find criteria.

r(allbut) all exclusion criteria.

Matrices

r(key) (m+1) x 6 matrix containing all find statistics.

r(but) [m,m+1] x 6 matrix containing all exclusion
statistics.

Author

Andre Assumpcao

The University of North Carolina at Chapel Hill

Department of Public Policy

aassumpcao@unc.edu

Acknowledgments

Cox (2011)
created the original number of occurrences statistics in textfind. Here I
have only modified the function arguments to allow for Unicode encoding search.

References

Cox, N. J. 2011. Stata tip 98: Counting substrings within strings. Stata Journal, 11(2): 318-320.

Also see

Help: [D] ustrregexm(), string functions, moss()

FAQs: What are regular expressions and how can I use them in Stata?

FAQs: How can I extract a portion of a string variable using regular expressions? | Stata FAQ