*! version 1.1 04feb2016
help for whichencodingversion 1.1 (04 February 2016)
help for ascii2unicodeversion 1.1 (04 February 2016)
help for unicode2asciiversion 1.1 (04 February 2016)

Title

whichencoding -- Examine the encoding of Stata datasets and
text files. Stata 14.1 (28jan2016 update) or newer required

ascii2unicode -- Translate datasets and text files from ASCII
to Unicode encoding. Stata 14.1 (28jan2016 update) or newer required

unicode2ascii -- Translate datasets and text files from Unicode
to ASCII encoding. Stata 14.1 (28jan2016 update) or newer required

Syntax

whichencoding filespec [ , detail nodata]

ascii2unicode filespec [ , encoding() suffix()
detail nodata replace]

unicode2ascii filespec [ , encoding() version()
suffix() detail nodata replace]

filespec specifies the file(s) to be examined or translated, typically
datasets and do-files.
You can specify single files or groups of files, for
example, *.dta *.do indicating all datasets and do-files in the current
directory.

Windows users: Windows is not case-sensitive concerning filenames, but the
unicode commands are, also in Stata for Windows, and this affects the
commands described here. For example, the file specifications b*.dta and
B*.dta select different datasets.

Main
encoding() Extended ASCII encoding scheme for ASCII source or
destination files. If not specified as recorded in the global macro
UnicodeEncoding; otherwise Windows-1252

version(#) Destination dataset version 12 or 13;
version 13 if not specified

suffix() Add a suffix to ASCII source or destination files;
_v12 or _v13 for datasets and _asc for text files if not specified

detail Display details about encoding of each file

nodata Do not analyze or translate contents of string variables

replace Overwrite existing destination files

Description

Stata version 14 uses Unicode encoding, and prior versions use ASCII
encoding. If you use plain ASCII characters only, for example if you use English
language and avoid extended ASCII characters like é, ü, and ñ, you need not
read further. Also, if you do not care if single characters are displayed
incorrectly, you might decide not to spend more time with this issue.
Read more about ASCII and Unicode encoding in the Remarks section.

If a Stata dataset or text file generated by Stata <14 contains extended ASCII
characters (for example, é, ü, ñ), Stata 14+ can open the file, but the extended
ASCII characters will not be displayed correctly. Similarly, if a dataset or
text file generated by Stata version 14+ contains Unicode encoded characters
beyond plain ASCII, you can use saveold to generate a file that can be
used by Stata 11-13, but the Unicode characters will not be displayed correctly.

whichencoding examines the occurrence of Unicode and extended ASCII
characters in Stata datasets and text files like do-files, ado-files, help files
and log files. This is useful to determine the need for translation when sharing
Stata files between users or computers with different versions of Stata
installed. The official unicode analyze command serves the same purpose,
but the output from whichencoding is more compact and transparent.

ascii2unicode translates datasets and text files with extended ASCII
characters to Unicode encoding. Files with plain ASCII characters only are not
translated. The destination file takes the name of the source
file, and a suffix is added to the source file name. The official
unicode translate serves the same purpose, but the output from
ascii2unicode is more compact and transparent, and you have access both to
a Unicode and an ASCII version of a dataset at the same time.

unicode2ascii translates datasets and text files with Unicode
characters to ASCII encoding. Files with plain ASCII characters only are not
translated. In datasets, variable names, labels and label names (including labels
in different languages), string variable contents, and notes are translated.
The source file keeps its name, and a suffix is added to the destination file name.
Currently (September 2015), no official Stata command serves the same purpose.

Options

Commands and their options

Command

Option whichencoding ascii2unicode unicode2ascii

encoding() + +
version() +
suffix() + +
detail + + +
nodata + + +
replace + +

encoding(encoding scheme) specifies the extended ASCII encoding
scheme for ASCII encoded source or destination files.
If the computer's encoding scheme is Latin 2,
make it the default by including this command in profile.do:

if c(stata_version)>=14 unicode encoding set Latin 2

This will create the global macro UnicodeEncoding, making
Latin 2 the default encoding scheme. If the macro is not defined,
the default encoding scheme will be Windows-1252.

(Commands in profile.do will be executed automatically at Stata
start-up. Use the sysdir command to locate the PERSONAL folder and
put profile.do there.)

version(#) (unicode2ascii) specifies the Stata version (12 or 13)
of the destination dataset; version 13 is the default. (The diligent user may
have noticed that Stata 14's help for saveold claims that saveold
can also save a version 11 dataset. This is not quite accurate: It can save a
version 12 dataset, which Stata 11 can read, but not a version 11 dataset,
which Stata 10 would be able to read.)

suffix() specifies a suffix to be included in ASCII destination file
names (unicode2ascii) or in modified ASCII source file names
(ascii2unicode). Default suffixes are _v12 or _v13 for datasets
and _asc for text files.

detail displays details about each file to be analyzed or translated. It
displays the output from unicode analyze, which can be quite verbose, so
avoid using this option for more than a few files.

nodata specifies that the contents of string variables should not be
analyzed or translated.

replace allows the destination dataset to overwrite an existing dataset
with the same name.

Examples: whichencoding

. whichencoding filea.dta , detail

. whichencoding *.dta *.do

We examine three datasets and three do-files. Two files contain plain
ASCII characters only, two files contain extended ASCII encoded characters,
and two files contain Unicode encoded characters. Here is the output:

Directory: C:\Docs\Project D

File name Version Encoding

filea.dta v13 Plain ASCII
fileb.dta v12 Extended ASCII
filec.dta v14 Unicode
gen_filea.do Plain ASCII
gen_fileb.do Extended ASCII
gen_filec.do Unicode

The output displays the version of datasets. We also
learn that filea.dta and gen_filea.do contain plain ASCII characters
only, so both Stata 13 and 14 can use them without any problems.

fileb.dta and gen_fileb.do contain extended ASCII characters.
They can be opened by Stata 14, but some characters will not be displayed
correctly unless they are translated to Unicode with the official
unicode translate command or the unofficial ascii2unicode
command (see below).

filec.dta and gen_filec.do contain Unicode encoded characters.
If needed, we can use saveold to generate a version that Stata 11, 12, or
13 can open, but some characters will not be displayed correctly.
Currently (February 2016), no official commands translate datasets from
Unicode to extended ASCII, but the unofficial unicode2ascii command
does (see below).

Examples: ascii2unicode

. ascii2unicode alpha.dta

. ascii2unicode *.dta *.do , encoding(Latin 2)

. ascii2unicode beta2.dta gen_beta2.do , detail replace

We examine six files, filea.dta and gen_filea.do, which contain
plain ASCII characters only, fileb.dta and gen_fileb.do, which contain
some extended ASCII charaters, and filec.dta and gen_filec.do,
which are Unicode encoded. The files with plain ASCII characters and the Unicode
encoded files need no translation, only the files with extended ASCII characters
are translated. Here is the output:

Directory: C:\docs\project D

Source files, modified names Result files, Unicode compatible

filea.dta = filea.dta
fileb_v12.dta --> fileb.dta
filec.dta = filec.dta
gen_filea.do = gen_filea.do
gen_fileb_asc.do --> gen_fileb.do
gen_filec.do = gen_filec.do

--> File translated; new source file name = File not translated

The result files in the rightmost column have the same names as the
original source files; four of them are actually the same files, but two files
were translated from ASCII to Unicode endoding. The original source files for
these two files had their names changed by including a suffix.
fileb.dta was a Stata version 12 dataset, hence the new name
fileb_v12.dta. For do-files and other text files the default suffix is
_asc; unlike datasets, they do not belong to a specifc Stata version.

Examples: unicode2ascii

. unicode2ascii alpha.dta

. unicode2ascii *.dta *.do , encoding(Latin 2)

. unicode2ascii beta2.dta gen_beta2.do , version(12) replace

We examine three datasets and three do-files to be used by Stata version 13, and
translate them from Unicode to ASCII, if necessary. This is the output:

Directory: C:\Docs\Project C

Source files Result files, ASCII compatible

v13: filea.dta = filea.dta
v12: fileb.dta = fileb.dta
v14: filec.dta --> filec_v13.dta
asc: gen_filea.do = gen_filea.do
asc: gen_fileb.do = gen_fileb.do
unc: gen_filec.do --> gen_filec_asc.do

--> File translated; new result file name = File not translated

The first column lists the source files, the second column the result files.
Source dataset filenames are preceded by their version, and do-files and
other text files are preceded by "unc" or "asc" to indicate their
encoding. If a translation is indicated, a suffix is added to the result file
name.

Remarks

Prior to version 14, Stata used ASCII encoding of characters.
The codes for plain ASCII are 0-127; for extended ASCII they are 128-255.
There are several extended ASCII encoding schemes, for example, Latin 1 and
Windows-1252 for Western European languages, Latin 2 for some Central
and Eastern European languages, and Latin 4 using the Cyrillic alphabet.
Thus, the same extended ASCII code may display different characters
by computers using different encoding schemes.

From version 14, Stata uses Unicode encoding of characters. This gives
access to thousands of characters and symbols, including Arabic, Cyrillic,
Chinese, and other alphabets. The Unicode code point is the number to use with
the uchar() function, but behind it is a more complex encoding (UTF-8)
where each character is defined by one to four 8-bit bytes. For the characters
represented in plain ASCII, the Unicode code and the ASCII code are the same,
and in the Latin 1 extended ASCII encoding, the Unicode code point
is the same as the ASCII code.

whichencoding, ascii2unicode, and unicode2ascii
utilize the official unicode analyze command. Currently (February 2016)
unicode analyze avoids analyzing a file if a backup file indicates that a
file with the same name has been analyzed previously. To make sure that revised
versions of already analyzed files are re-analyzed, whichencoding and
ascii2unicode rename the backup file for the file being analyzed.
Backup files are located in the bak.stunicode and
bak.stunicode/status.stunicode subdirectories to the current directory.

Authors

Svend Juul

Aarhus University

sj@ph.au.dk

Morten Frydenberg

Aarhus University

morten@ph.au.dk

Also see

help for unicode (Stata 14+ only)

help for saveold