*! version 1.0 17june2026

Title

littext -- Automated construct discovery and relationship
inference from an academic text

The short alias litt is provided for interactive use; the two commands
are functionally identical.

Syntax

littext analyze , text(varname) [ options ]

littext graph , [ type(string) top(#)
outdir(string) weighted level(string)
format(string) embed(string) saving(string) replace ]

littext export , outdir(string) [ name(string)
format(string) minconf(#) type(string) top(#)
columns(string) ]

littext example [ , clear ]

littext install [ , quiet verbose ]

Description

littext extracts candidate construct relationships from an unstructured
corpus of academic text (titles, abstracts, full texts) or other research
text such as interview transcripts, consumer reviews, and social media
comments. It is intended for the exploratory researcher who has assembled a
large corpus and wants to generate candidate relationships of the form
"X is associated with Y", "X moderates the effect of Z on Y", etc., that
can then be hand-curated into a formal systematic literature review coding
scheme.

The pipeline is: text-kind-appropriate cleaning; spaCy noun-chunk
extraction; sentence-transformer embedding of candidate constructs;
HDBSCAN clustering into synonym groups; lexical construct-hierarchy
detection; co-occurrence-based relation candidacy with normalized
PMI scoring; dependency-pattern matching for relationship valence.

Results are returned in three Stata frames left in memory:

lt_constructs - one row per extracted construct

lt_relations - one row per candidate relationship

lt_diag - one row per source document with diagnostics

After littext analyze returns, a user remains in the frame it called
from; the results are in the named frames lt_constructs,
lt_relations, and lt_diag, queried with a frame prefix, e.g.
frame lt_relations: list source target relation_type confidence.
Files on disk are produced only if you pass saving().

Options for littext analyze

text(varname) (required) - the variable in the current dataset that
holds the document text. May be a str# or strL variable. Must be
a string variable; numeric variables are rejected with an error.

id(varname) - a per-document identifier. If omitted, _n is used.

year(varname) - publication year (numeric). Used for the trend graph
and stored in lt_diag.

journal(varname) - outlet name (string). Stored in lt_diag for
comparative analysis.

texttype(string) - declares the kind of text in the corpus. One of:

abstract - academic abstracts (default; Emerald and copyright cleaners)

fulltext - full papers (above plus LaTeX, references, captions)

transcript - interview / focus group transcripts (speaker labels, timestamps)

review - consumer reviews (HTML, ratings, verified-purchase labels)

comment - social-media comments (URLs; emoticons preserved)

other - minimal cleaning only (whitespace, control chars)

If texttype() is not declared, the package defaults to
abstract and emits a note indicating that this default was applied.
The declaration drives the cleaning regime, the default unit(),
and the default mintextlen(). A post-clean median-length sanity
check warns when the corpus length is outside the typical window for
the declared texttype, which most often indicates a misdeclared
text() variable.

unit(string) - unit of analysis for relationship candidacy. One of
sentence, abstract, paragraph. If not specified, defaults
from texttype(): sentence for abstract/transcript/review/comment/other,
paragraph for fulltext.

embedmodel(string) - name of the sentence-transformers model used for
construct embeddings.

minfreq(#) - minimum document frequency for a candidate construct to
be retained. Default: 1 for corpora with fewer than 50 documents,
2 otherwise. Resolved value and rationale are printed at run time.

maxrelations(#) - cap on the number of candidate relationships
written to lt_relations (highest-confidence first). Default 100000.

mintextlen(#) - minimum text length in characters. Rows whose
text() value is shorter than this threshold are dropped before the
pipeline runs. If not specified, defaults from texttype(): 50 for
abstract/other, 500 for fulltext, 30 for transcript, 20 for review,
10 for comment. Pass keepempty to disable row-dropping entirely.

keepempty - retain all rows including empty, whitespace-only, and
below-threshold ones. The default behavior is to drop these with a logged
count. Use this when the corpus is being analyzed for a purpose that
requires preserving the input row count.

addsentiment - additionally compute VADER affective polarity on each
evidence sentence and store it in text_polarity. Note: this is
affective sentiment of the text, NOT relationship valence.
Relationship valence is always computed and stored in relation_type.

quiet - suppress progress output.

saving(string) - if specified, the three frames are also saved as
stub_constructs.dta, stub_relations.dta, stub_diag.dta.

replace - allow overwriting existing files when saving() is used.

Options for littext graph

type(string) - figure type. One of:

frequency - bar chart of top-k constructs (Stata-native)

distribution - distribution of relation types (Stata-native)

trend - extraction yield over years (Stata-native)

confidence - histogram of confidence scores (Stata-native)

extraction - distribution by extraction method (Stata-native)

map - UMAP concept map (matplotlib; default)

network - relationship network (matplotlib)

dendrogram - construct-cluster dendrogram (matplotlib)

cooccurrence - pairwise NPMI heatmap of top-k constructs (matplotlib)

roles - construct x relation-type heatmap (matplotlib)

top(#) - number of top constructs or relationships to display.
Default 20. For heatmaps, controls the matrix dimensions.

weighted - for type(network) only: color edges continuously by
confidence (viridis) rather than discretely by relation type. Useful when
edge strength matters more than syntactic type.

level(string) - hierarchy specificity for construct-vocabulary graph
types. Accepts leaf (default; constructs at maximum specificity),
root (each construct replaced by its hierarchy root), or a
non-negative integer N (collapse to depth N). Honoured by
type(frequency) and by the matplotlib type(map)
and type(network) renderers. In map a rolled construct is drawn
at the frequency-weighted centroid of its children. In network edges
are aggregated within each relation type but never across types: a
positive and a negative edge between the same rolled pair stay distinct.
Other Stata-native types, the heatmaps (cooccurrence, roles),
and dendrogram (whose tree is built from cluster distances, not the
construct hierarchy) ignore level() and emit a one-line note. The
hierarchy is computed by the lexical right-substring rule plus the
hyphenated-prefix rule described in the Notes section.

outdir(string) - directory where figure files will be written.
REQUIRED. Pass an absolute path (e.g. "D:\projects\figures"). If
omitted, littext graph stops with an error rather than guessing a
location. A relative path is accepted but resolved against the current
working directory (c(pwd)) with a warning. The resolved absolute
path is printed on every save.

saving(string) - output file stub for matplotlib figures (PNG and PDF
are written). For Stata-native graphs, the file is saved as PNG via
graph export.

format(string) - output format for the matplotlib figure types
(map, network, dendrogram, cooccurrence, roles).
Accepts static (default; PNG and PDF via matplotlib), html
(interactive Plotly HTML), or both. Ignored with a note for
Stata-native types, which are always static.

embed(string) - how plotly.js is embedded in format(html)
output. selfcontained (default) writes a standalone file (~3.5 MB)
that opens offline on any machine; cdn writes a small file that
loads plotly.js from a content-delivery network and therefore needs an
internet connection to render.

Options for littext export

littext export writes the candidate relationships from the most recent
littext analyze (the lt_relations frame) as a hypothesis register
for hand-curation: a clean candidate table, sorted strongest-first, with no
curation columns added (the analyst adds their own). Run littext
analyze first.

outdir(string) - REQUIRED. Absolute path to the directory where the
register is written. A relative path is resolved against c(pwd) with a
warning.

name(string) - file-name stub for the register (default
littext_register). The extension is added per format().

format(string) - output format: csv (default), xlsx, or
both. CSV is written with full quoting, so evidence spans containing
commas or quotes survive intact.

minconf(#) - keep only candidates with confidence at or above
this value (default: keep all).

type(string) - restrict to one or more relation types, given as a
space- or comma-separated list (e.g. type(pos_assoc neg_assoc)).

top(#) - keep only the top # candidates after sorting by
descending confidence (default: keep all).

columns(string) - space-separated list of lt_relations columns
to export. Unknown columns are skipped with a note.

Stata frames produced

lt_constructs: construct_id, surface_form, canonical_form, cluster_id,
freq_doc, freq_total, parent_canonical, canonical_root, hierarchy_depth,
is_root.

lt_relations: rel_id, doc_id, unit_id, source, target,
source_construct_id, target_construct_id, relation_type, confidence,
extraction_method, evidence_text, text_polarity.

lt_diag: doc_id, year, journal, n_constructs_extracted,
n_relations_extracted.

relation_type vocabulary

pos_assoc - positive association (X increases/enhances/predicts Y)

neg_assoc - negative association (X reduces/attenuates Y)

moderates - X moderates the relationship between two others

mediates - X mediates the effect of one construct on another

causes - X causes / leads to Y

assoc - non-directional or unclassified co-occurrence

Notes

row-drop behavior. By default, littext analyze drops rows
where the text() variable is missing or whitespace-only, rows where
a user-supplied id() variable is missing, and rows whose text is
shorter than mintextlen() characters. A summary of the drops is
printed (suppressed under quiet). A warning is emitted if more
than 25% of input rows are dropped, which most often indicates that the
text() variable points at the wrong column.

text-kind declaration. The texttype() option drives
three downstream defaults: which cleaning regime is applied to the raw
text; the default segmentation unit(); and the default
mintextlen(). Each derived default is overridable by passing the
corresponding option explicitly. The resolved values and their sources
("user-specified" vs "texttype default") are printed at run time.
A post-clean median-length sanity check warns when the corpus length
falls outside the typical window for the declared texttype.

construct hierarchy. Four new columns are added to
lt_constructs: parent_canonical (the canonical form of the
immediate IS-A parent, or empty if the construct is a root),
canonical_root (the topmost ancestor; equals canonical_form
for roots), hierarchy_depth (zero for roots, one for direct
children), and is_root (1 if root, 0 otherwise). The hierarchy is
detected by a lexical right-substring rule with a frequency prior,
supplemented by a hyphenated-prefix rule that admits constructs of the
form X-based Parent, X-driven Parent, X-led Parent, and
X-oriented Parent as children of Parent regardless of the
frequency prior. The rule is English-specific and is silent on
conceptually-subsumed but lexically-distinct relations (e.g., it does
not link brand reputation to brand equity because they
share no right substring).

Example hierarchies the rule recovers. For instance, if a corpus contains
brand equity and any of consumer-based brand equity,
financial-based brand equity, online brand equity, or
employee-based brand equity, the rule places each subtype as a
depth-1 child of brand equity. Query the hierarchy with:

. frame lt_constructs: list canonical_form parent_canonical canonical_root, sepby(canonical_root)

or roll up at the graph level with:

. littext graph, type(frequency) level(root)

Examples

Load the bundled synthetic RBV corpus (300 abstracts) and analyze it:

. littext example, clear

. littext analyze, text(abstract) id(article_id) year(year) journal(journal) texttype(abstract)

. frame lt_relations: list source target relation_type confidence in 1/10

. frame lt_relations: tab relation_type

. littext graph, type(map) outdir("D:/figs")

. littext graph, type(network) top(25)

. littext graph, type(network) level(root)

. littext graph, type(map) level(1)

. littext graph, type(network) outdir("D:/figs") format(html)

. littext graph, type(map) outdir("D:/figs") format(both)

. littext export, outdir("D:/register") format(both)

. littext export, outdir("D:/register") minconf(0.7) type(pos_assoc neg_assoc) top(200)

. littext graph, type(frequency) level(root)

For a corpus of interview transcripts:

. use my_transcripts.dta, clear

. littext analyze, text(transcript) id(case_id) texttype(transcript)

For a corpus of consumer reviews:

. use product_reviews.dta, clear

. littext analyze, text(review_text) id(review_id) texttype(review)

For full-text academic papers (LaTeX or PDF-extracted text):

. use my_fulltexts.dta, clear

. littext analyze, text(body) id(paper_id) texttype(fulltext)

Sentiment analysis: a note

littext draws a clear line between two distinct constructs that are often
conflated in social sciences/marketing/management applications:

1. Relationship valence is the sign of the directional relationship
between two constructs (X positively/negatively related to Y). This is always
computed and stored in relation_type. It is essential to the purpose of
the package; a hypothesis register that cannot distinguish "X increases Y"
from "X reduces Y" is not a hypothesis register.

2. Affective sentiment is the emotional polarity of a piece of text, in
the sense of VADER, LIWC, or the NRC Emotion Lexicon. This is meaningful for
consumer-text corpora (reviews, tweets) but largely uninformative for academic
abstracts. littext computes it only on request via addsentiment and
stores it in text_polarity.

Users should not treat text_polarity as a measure of relationship sign.

Requirements

Stata 19 or higher with Python integration configured. Python 3.14
recommended on Windows; spaCy on Python requires blis 1.3.3 or
higher. Required Python packages: spacy, sentence-transformers, hdbscan,
scikit-learn, umap-learn, matplotlib, networkx, plotly, pandas, numpy. The spaCy
model en_core_web_sm must be downloaded once via
python -m spacy download en_core_web_sm.

Limitations

littext uses noun-chunk extraction rather than a domain-trained NER
model, and co-occurrence plus dependency-pattern matching rather than a
trained relation extractor. It is therefore best understood as a
candidate-generation tool whose output requires manual curation before
being treated as a coding scheme. Quantitative precision/recall figures
should not be reported against the bundled synthetic corpus.

The construct-hierarchy detector and the texttype() cleaning
regimes are English-specific. The hierarchy rule does not recover
conceptual subsumption that lacks a lexical signal.

References

Bouma, G. (2009). Normalized (pointwise) mutual information in collocation
extraction. In Proceedings of GSCL, 30, 31-40.

Hearst, M. A. (1992). Automatic acquisition of hyponyms from large text
corpora. In Proceedings of COLING-92, 539-545.

Hutto, C. J., & Gilbert, E. (2014). VADER: A parsimonious rule-based model
for sentiment analysis of social media text. In Proceedings of ICWSM, 8(1), 216-225.

Li, J., Larsen, K. R., & Abbasi, A. (2020). TheoryOn: A design framework and
system for unlocking behavioral knowledge through ontology learning. MIS Quarterly, 44(4), 1733-1772.

Aliases

litt is provided as a short-form alias for littext. It
forwards every argument and propagates returned scalars and macros.

Author

Nebojsa S. Davcik

EM Normandie Business School, Oxford, UK

ORCID: 0000-0003-1041-8788

https://orcid.org/0000-0003-1041-8788

Email: davcik@live.com

Citation

When citing littext in academic work, please use:

Davcik, N. S. 2026. LITTEXT: Stata module for automated construct discovery and relationship inference from academic text.
Available at: https://github.com/Davcik/littext

License

littext is free software released under the
GNU General Public License version 3 or later (GPL-3.0-or-later).
You may redistribute and modify it under the terms of that license;
modified versions and larger works that incorporate littext
must also be released under GPL-3 or later. See the LICENSE file in
the repository root for the full license text.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Also see

The command: littext