*! version 1.0.0 17feb2019

Title

chartab -- Table of Unicode character frequency counts

Syntax

chartab [varlist] [if] [in] [, options]

files(filename:filenames) list of file names (or URLs) to process

scalars(scalar:string_scalars) list of string scalars to process

literals(strings:string_literals) list of string literals to process

noascii do not tally or report frequency counts for ASCII characters

replace replace the data in memory with tabulation results

store create stored results for each Unicode character observed

ucd(filename:UCDfilename) specify a custom UCD (Unicode Character Database)

Description

The chartab command
(requires Stata version 14 or higher)
produces a table of Unicode character frequency counts.
The companion chartabb command
(requires Stata version 10 or higher)
produces a table of byte code frequency counts.

chartab can process any combination of
string variables,
files,
string scalars, and
string literals in a single run.
The table produced reports the Unicode character frequency counts
from all sources combined.

Before Stata version 14,
a character is encoded using a single byte.
This allows for 256 distinct values.
The first 128 byte codes (values 0 to 127) are ASCII codes.
There is no standard for the remaining byte codes (values 128 to 255).

Starting with Stata version 14,
Stata operates as if all characters are encoded in UTF-8.
This is a storage-efficient Unicode encoding where ASCII characters
are encoded using a single byte
(using the same ASCII byte code).
All other Unicode characters are encoded using a
multi-byte sequence (from two to four bytes,
each with a value greater than 127).
So by design, UTF-8 is backwards compatible with ASCII.

chartab produces a table that includes,
for each Unicode character found in the source,
the decimal value of the code point (which can be used
with Stata's uchar() function),
its hexadecimal string representation in UTF-8
(which can be used with the ustrunescape() function),
the glyph (the visual representation of the character),
the frequency count,
and finally the Unicode unique name of the character.
The table is by default printed in the Results window
but you can instead replace the data in memory with the results
if you specify the replace option.

chartab processes ASCII and multi-byte UTF-8 sequences separately.
If you are not interested in frequency counts for ASCII characters,
you can specify the noascii option and chartab may run
a bit faster.

If you feed chartab text that is not encoded in UTF-8,
each invalid UTF-8 byte sequence will be replaced
by a Unicode Replacement Character (uchar(65533) or \ufffd in hex).
See ustrfix().
Stata has Unicode utilities to translate text to UTF-8.

Options

Options

files(filename:filenames) specifies a list of files to process.
You can also specify URLs instead of a filenames.

scalars(scalar:string_scalars) specifies a list of string
scalar names to process.

literals(strings:string_literals) specifies a list of string
literals to process.

noascii if you do not want to tabulate ASCII characters.

replace to replace the data in memory with the tabulation results
(and skip printing the table in the Results window).

store to store the frequency count for each Unicode code point observed
in the source.

ucd(filename:UCDfilename) to supply a custom UCD.
A sizable portion of the Unicode space is reserved for private-use characters.
This option can be used to associate unique names for
these private-use characters.

Examples

String data encoded in ASCII can be used with Unicode Stata without translation
because single-byte UTF-8 encodings match ASCII codes:

example do-file content

version 14

sysuse auto, clear
chartab make

(click to run)

You can use chartab with string scalars and string literals:

example do-file content

version 14

scalar s = "Citroën C–Elysée II"
chartab , scalar(s)

chartab , literal("Citroën C–Elysée II")

(click to run)

chartab can also process files.
Use the noascii option if you are only interested in multi-byte
UTF-8 characters:

example do-file content

version 14

sysuse auto, clear
replace make = "Citroën C–Elysée II" in 64

tempfile f
export delimited make using "`f'"

clear
chartab, file("`f'") noascii

(click to run)

You can also process a URL as if it were a local file.
In the following example, chartab will process this
UTF-8 sample web page.
There are a lot of Unicode characters so we use the replace option.
With the results in memory, we use Stata's ustrto() function to convert
each Unicode character to ASCII.
This can be useful to remove accents from letters but notice the last observation
listed.
You can browse the results to see that most of the Unicode
characters have no suitable equivalent in ASCII.

example do-file content

version 14

// the original UTF-8 plain-text file is encapsulated in HTML
type "https://www.w3.org/2001/06/utf-8-test/UTF-8-demo.html"

chartab, file("https://www.w3.org/2001/06/utf-8-test/UTF-8-demo.html") replace

// convert each character to ASCII
gen c = ustrto(ustrnormalize(cp_char, "nfd"), "ascii", 2)
list if cp_dec > 127 & c != ""

(click to run)

Stored results

chartab stores the following in r():

Scalars

r(fc) number of Unicode characters

r(distinct) number of distinct Unicode characters

r(fc_ascii) number of ASCII characters

r(dis_ascii) number of distinct ASCII characters

r(fc_mb_utf8) number of multi-byte Unicode characters

r(dis_mb_utf8) number of distinct multi-byte Unicode characters

r(urc) number of Unicode Replacement Character

r(has_urc) 0 if the Unicode Replacement Character is not observed, 1 otherwise

r(ucdsize) number of code points in the repertoire of the supplied UCD (Unicode Character Database)

The Unicode code space spans 17 contiguous planes, each with 16-bit code points
(17 * 2^16 == 1,114,112 code points).
Some code points cannot be encoded in UTF-8.
When the store option is specified, chartab also stores in r()
the frequency count for all UTF-8 characters observed in the source:

Scalars

r(c0) number of uchar(0)

r(c1) number of uchar(1)

r(c2) number of uchar(2)

... ...

r(c1114111) number of uchar(1114111)

macros

r(ascii_printable) list of printable ASCII code points observed

r(ascii_control) list of ASCII control code points observed

r(multibyte) list of multi-byte UTF-8 code points observed

References

The Unicode Consortium.
The Unicode Standard, Version 11.0.0,
(Mountain View, CA: The Unicode Consortium, 2018. ISBN 978-1-936213-19-1).

Unicode Standard Annex #42: Unicode Character Database in XML.

Unicode Standard Annex #44: Unicode Character Database.

Flat version of the UCD:
https://www.unicode.org/Public/UCD/latest/ucdxml/ucd.all.flat.zip

SIL International, NRSI: Computers & Writing Systems,
Mapping codepoints to Unicode encoding forms.

Wikipedia,
UTF-8.

License to redistribute the UCD

The chartab package includes a subset of the UCD (Unicode Character Database).
Pursuant to the Unicode, Inc. License Agreement,
the copyright and permission notice appears at the end of
the chartab ado-file.

Author

Robert Picard

robertpicard@gmail.com