*! version 1.1.2 07oct2024

Title

usort -- byable locale-based ascending and descending sort that
supports conditional statements, observation ranges, and user-defined handling
of substrings and missing values

Syntax

usort
[+|-]
varname
[[+|-]
varname ...]
[if] [in]
[, options]

Substrings and missing values
first(strings:string [, pos|rpos|regex]]) sort the words, i.e.,
substrings (not) enclosed in double quotes and separated by
whitespace(s), within the string: first (for
[+]varname) or last (for -varname) in the specified order (word 1
is first, word 2 is second, etc.). Sorting is done by: a)
comparing varname with each word (without options), b) using
ustrpos(varname, word) (for pos), c) using
ustrrpos(varname, word) (for rpos), or
d) using ustrregexm(varname, word) (for
regex).
Note: System numerical and string missing
values are coded as ., while non-system missing values are coded as
.a, .b, etc. in non-regex representation. For regular
expressions, you can use ^[.]$, ^[.]a$, ^[.]b$, etc. in
first(string, regex).

last(strings:string [, pos|rpos|regex]]) sort the words, i.e.,
substrings not enclosed in double quotes and separated by
whitespace(s), within the string: last (for
[+]varname) or first (for -varname) in the specified order (word 1
is last, word 2 is second to last, etc.). Sorting is performed by: a)
comparing varname with each word (without options), b) using
ustrpos(varname, word) (for pos), c) using
ustrrpos(varname, word) (for rpos), or
d) using ustrregexm(varname, word) (for
regex).
Note: System numerical and string missing
values are coded as ., while non-system missing values are coded as
.a, .b, etc. in non-regex representation. For regular
expressions, you can use ^[.]$, ^[.]a$, ^[.]b$, etc. in
first(string, regex).

ignorec ignore case sensitivity in first() and last().

mfirst sort missing values first (for [+]varname) or last (for
-varname).

mlast sort missing values last (for [+]varname) or first (for
-varname).

ignorem ignore missing values when using first(),
last(), mfirst, and mlast.

Locale
locale(string) locale code from the unicode locale list
or c(locale_functions) by default.

st(#) argument st in
ustrsortkeyex(), with a default value of -1.

case(#) argument case in
ustrsortkeyex(), with a default value of -1.

cslv(#) argument cslv in
ustrsortkeyex(), with a default value of -1.

norm(#) argument norm in
ustrsortkeyex(), with a default value of -1.

num(#) argument num in
ustrsortkeyex(), with a default value of -1.

alt(#) argument alt in
ustrsortkeyex(), with a default value of -1.

fr(#) argument fr in
ustrsortkeyex(), with a default value of -1.

Miscellaneous
format(%fmt) format for converting numerical sort variables into
strings (sorting is performed on string values only). The default format
is %32.16f.

codepoint(#) code point location of a symbol from the bottom of
the UTF-8 table used to make last() work. The default value is
129769.

by is allowed; see by.

weights are not allowed; see weights.

Description

This program is a byable sort command, which allows for: a) custom first and
last substrings, including system missing values (.) and all other missing
values, b) gsort-like syntax for sorting in ascending or descending
order, and c) conditional sorting using [if] or range-based
sorting using [in]. The program is built around Stata's
sort command and will mark the dataset as sorted (sorted by) if all rows
are selected. If a subset of rows is selected, it applies Mata's
_collate().

Sorting large datasets may be more taxing on machine CPU, memory, and/or disk
space as compared to sort and gsort.

Methods and Formulas

Sorting occurs in two steps:

1. Generating a permutation vector in Mata from the sort
variables under preserve. Since non-numeric sorting values cannot be
'destringed', the sort variable type must be str#/strL
to allow sorting them as a single matrix using Mata's
sort() function. The precision for sorting 'tostringed' numeric
values is determined by the %fmt (either default or
user-specified) in format().

To ensure that substrings specified by the first() option are sorted first,
they are replaced within the sort variables by " #", where " " is a
string of whitespaces (a Unicode character from the top of the UTF-8 table) with
a length of max(strlen(sort variable)). This step is skipped for already
'tostringed' missing values (., .a, ..., .z) if ignorem is
specified.

To ensure that substrings specified by the last() option are sorted last,
they are replaced within the sort variables by "©#", where "©" is a
string of identical Unicode characters from the bottom of the UTF-8 table. The
code point for this character (either default or user-specified) is set by
codepoint(), and the length is again max(strlen(sort variable)). This step
is also skipped for 'tostringed' missing values if ignorem is specified.

For natural sorting, leading zeros are appended to the integer parts of
'tostringed' numeric values.

2. Collating all rows with or a subset without adding the data-sorted flag
(sorted by) using the permutation vector.

The flag is created by preserving the original string and numeric values of the
sort variables in two ancillary matrices in Mata, replacing them
with the permutation vector, performing the regular Stata sort (i.e.,
reordering and collating), and then restoring the original sort variable values,
now collated on the permutation vector, using Mata's
_collate().

The program sets a data-changed flag when variable rows are collated.

Note: The by prefix is processed using egen group,
in conjunction with preserve, append, and save, which
store interim results in a temporary file.

Examples

Setup:

. sysuse auto

Sort observations in ascending order by price:

. usort price

Sort observations in ascending order by rep78, missing first:

. usort rep78, mfirst

Sort observations in ascending order by make in Czech, grouped by
foreign, with VW models placed at the top:

. bysort foreign: usort make, first(VW, pos) loc(cs_CS)

Sort observations in descending order by mpg and price:

. usort -mpg -price

Sort observations in descending order by price for domestic cars
only:

. usort -mpg -price if ! foreign

Acknowledgements

A special thanks to Leonardo Guizzetti for requesting and testing this program.

Author

Ilya Bolotov

Prague University of Economics and Business

Prague, Czech Republic

ilya.bolotov@vse.cz

Thanks for citing this software and my works on the topic:

Bolotov, I. (2024). USORT: Stata module to perform locale-based ascending
and descending sort that supports conditional statements, observation
ranges, and user-defined handling of substrings and missing values.
Available from https://ideas.repec.org/c/boc/bocode/s459385.html.