*! version 0.0.3 3mar2017

Title

ngram -- n-gram feature extractor

Syntax

ngram varname [if] [in] [, options]

degree(ngram##degree:#) The degree of n-grams to extract. Default: degree(1).

threshold(ngram##threshold:#) The minimum number of occurrences required across all observations to count a particular n-gram. Default: threshold(5).

prefix(ngram##prefix:prefix) A string to prepend to output n-gram columns. Default: prefix(t_).

binarize Instead of counting occurrences, only report whether or not each n-gram appeared in the text, 1 for appeared and 0 for did not appear. Default: disabled.

locale(ngram##locale:locale) The locale of the text.
Default: read from c(locale_functions). Avoid direct use: this only exists for pre-Stata 14 compatibility.

stopwords(ngram##stopwords:...) List of words to filter out before counting n-grams. Default: loaded from a list determined by locale, if found, else empty.

stemmer Whether to normalize words with the stemmer for locale.
Stemmers are available for da, de, en, es, fr, it, nl, no, pt, ro, ru, sv. Default: disabled.

lower Whether to normalize words by lower-casing them.
Default: enabled. Disable with nolower.

punctuation Whether to consider punctuation (as defined by locale) as delimiters. Default: disabled.

n_token Whether to report the number of tokens in each text
in a column named n_token. Default: enabled. Disable with non_token.

varname must be a string variable.

Description

ngram extracts n-gram counts, for text-mining applications, from string variable varname.

An n-gram is an n-long sequence of words.
For example, "sheep" is a unigram (1-gram), "black sheep" is a bigram (2-gram),
and "the black sheep is happy" is a 5-gram, although by default ngram's stopwords
will filter this to the trigram "black sheep happy".
What counts as a word to ngram is defined by the set of active delimiters.
To this command, a word is a sequence of characters separated by whitespace and/or punctuation; some applications
--- for example DNA mining and password generation --- build n-grams from other kinds of tokens,
like single characters or single syllables,
but this is not directly possible with this command.

n-gram counts can be used to produce markov models which emulate speech,
such as those used in autocorrect software,
can be used as features in machine learning / data mining algorithms,
such as sentiment analysis.
For a survey of the theory see Desolneux (2010), Chapter 1.
For a practical introduction to text-mining,
though unfortunately grounded in a non-Stata platform,
see Bird (2009).

ngram finds all n-grams in each given piece of text and counts them.
Each distinct n-gram is recorded in a new column, named after it; for example,
"black sheep happy" will be counted in a column named "t_black_sheep_happy"
(though the "t_" part can be changed if needed).

As a special feature, ngram adds special tokens from the ASCII table
STX and ETX to mark the start and end of the text.
These are reported as part of multigrams, so "we looked in the empty box."
will produce, in addition to bigrams "we looked" and "empty box",
the trigrams "STX we looked" and "empty box ETX".

Remarks related to LINUX

In Linux, when setting a locale (set locale_functions) other than your default locale
you may get an error asserting that the locale functions is not installed.
This is an LINUX problem. For the locale "de", for example, this can be remedied as follows:

sudo locale-gen "de_DE.UTF-8"

sudo dpkg-reconfigure locales

Options

degree determines the size of n-grams to extract.
All n-grams from degree 1 (unigrams) through degree (degree-grams) are extracted.

threshold specifies the rarity threshold required to report an n-grams.
An n-gram must appear in threshold distinct observations to become a part of the output table.

For example, suppose threshold(10), and a single respondent is very enthusiastic
about their "sheep dog", say, mentioning it sixteen times, and on the other hand
sixteen different texts mention "potato pie" once or twice; "sheep dog" would
count as rare and be ignored and "potato pie" would be counted as common and included.

This option drastically affects the size of the output, making the usually sparse dataset denser.

prefix gives a string which is prepended to n-grams before they are made into columns.
For example, "sheep dog" by default becomes "t_sheep_dog" and the processed text columns can referenced by "t_*".
This option lets you change the default prefix in the rare cases that need to avoid name collisions.

binarize instructs the algorithm to only report whether or not each n-gram
appeared in each text, instead of counting occurrences.
1 means appeared and 0 means did not.
This loses fidelity, but makes the dataset simpler for data-mining algorithms
like svm to handle.

locale gives the ICU locale of the text.
This determines the default stopwords, default stemming algorithm, and the way characters are tokenized.
Stata gained internal Unicode support via libicu in Stata 14,
and in Stata 14 and up you should specify the locale near the beginning of your script
with set locale_functions.
This option only exists for pre-Stata 14 compatibility.

However, in order to make use of ngram with foreign languages ---
that is, any language which your operating system is not set to ---
you need to be aware of how to write a locale.

Locale strings generally consist of an ISO language code and an optional ISO region code,
given as "lang" or "lang_Region". For example, "nl" is Dutch and "zh_TW" is Taiwanese Chinese.

On Stata 14 and up you can see the full list with unicode locale list.

stopwords is a whitespace-separated list of words to filter out before counting n-grams.
The intended purpose of stopwords is remove common words like prepositions and pronouns
which do not carry global meaning and just distract from the more meaningful nouns and verbs.

If not specified, the default stopwords are loaded from a file of
whitespace-separated words on the adopath---in particular, it can be in the working directory---
named "stopwords_la.txt" where "la" is the 2-letter language code of the current locale.
If the current language has no default file, a warning will be printed and no stopwords will be used, but the algorithm will still run.
Specifying stopwords() is equivalent to not specifying anything.

If stopwords(.) is specified, no stopwords will be used (this is different than stopwords()).

It is possible to add additional stopwords to the default file by specifying + as the first word of stopwords.
For example, stopwords("+ jane austen") appends these two additional words to the stopword file.

Stopwords are always case-insensitive.

The default stopword files that come with the distribution can be overridden by creating a different version of that file in the working directory.
For example, the stopword files for English, Dutch and German are called
"stopwords_en.txt", "stopwords_nl.txt" and "stopwords_de.txt", respectively. If placed in the working directory, the default
stopword files supplied will be ignored.

stemmer instructs the tokenizer to attempt to construct the root of the word (the "stem").
The stemmer attempts to remove the prefixes, suffixes, and other transformations
that decline a root word into adjective, verb, or noun forms.

Which stemmer to use is determined by the language code in locale.

Stemming for the following locales are supported:
da (Danish), de (German), en (English), es (Spanish), fr (French),
it (Italian), nl (Dutch), no (Norwegian), pt (Portuguese), ro (Romanian),
ru (Russian), and sv (Swedish).
There is no user-level way to add more, unfortunately.
If your dataset is not in one of these languages, you cannot use this option.

lower instructs the tokenizer to normalize words by lower-casing them.
This has no effect for languages without the upper/lower case distinction.

punctuation and instructs the tokenizer to include punctuation, as defined by locale, as tokens.

n_token instructs ngram to report the number of tokens in each text in a column named n_token.
This is the pre-normalizing pre-stopwords count of tokens, which is also the number of unigrams.

Since this always produces a column named n_token, you must rename any pre-existing column with that name first, or disable this with non_token.

Stored results

r(words) String containing all ngram variables created

Examples

Remarks

Control characters: control characters like ESC, ACK, Backspace are always considered delimiters by the tokenizer.
This avoids conflict with the STX/ETX feature.
If you do need to tokenize these characters, preprocess your text
replacing control characters by unique tokens of your own choosing.

Copyright

Copyright 2014-2017 Matthias Schonlau

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

For the GNU General Public License see <http://www.gnu.org/licenses/>.

Authors

Matthias Schonlau <schonlau@uwaterloo.ca>

Nick Guenther <nguenthe@uwaterloo.ca>

Joey Zhao <y266zhao@uwaterloo.ca>

Ilia Sucholutsky <isucholu@uwaterloo.ca>

References

Desolneux, Agnès. Pattern Theory.
CRC Press, 2010.
http://dx.doi.org/10.1201/b10620

Bird, Steven, Ewan Klein, and Edward Loper.
Natural Language Processing with Python.
O'Reilly, 2009.
http://www.nltk.org/book/