help for zipffitAlexander Koplenig (July 2014)

Fitting a Zipf or a Zipf-Mandelbrot distribution by ML

zipffit var [if]
[,
zmandelbrot
rank(#)
log maximize_options]

by is allowed

Options

zmand specifies to fit the two parameters of the Zipf-Mandelbrot distribution.
By default, the parameter of the Zipf distribution is fitted.

rank(#) requests to fit the distribution only for the first # ranks.

log displays the iteration log. Suppressed by default.

All standard maximize_options are available (maximize). Using difficult or technique can sometimes help to achieve convergence,
when the message "not concave" appears repeatedly.

Description

zipffit fits a Zipf distribution or a Zipf-Mandelbrot distribution by ML using a right truncated zeta distribution.
If one -gsorts a list of observations by the frequency of var
and assign rank 1 to the most frequent observation,
rank 2 to the second most frequent observation, and so on,
then many empirical phenomena (e.g. word frequency distributions, city sizes, income distributions, relative species abundance)
tend to be Zipf distributed (Zipf, 1935), that is, they (roughly) obey the following relation
between the rank r of an observation and its frequency f_r:

f_r~r^(-a)

where a is a parameter that has to be determined empirically.
[NB.: if two observations have the same var frequency, then individual ranks are assigned randomly, that is,
ties are broken arbitrarily.]

Mathematically, Zipf’s law can be modeled as a right-truncated
zeta distribution (Baixeries et al. , 2013),
where the probability p of a word with rank r is given by

p_r=(1/H(N,a))*r^(-a)

where H(N,a) is defined as

H(N,a) = SUM(1/r^(a))

Here, the SUM is the sum starting from rank one to the maximum rank N, which is the observed number of observations (e.g. different word types)

Correspondingly, the Zipf-Mandelbrot law (Mandelbrot, 1953) can be modeled as

p_r=(1/H(N,a,ß))*(r+ß)^(-a)

with

H(N,a,ß) = SUM(1/(r+ß)^(a))

Since Zipf's law is just a special case of the Zipf-Mandelbrot law
with the second parameter ß set to 0, the following description focusses on the maximum likelihood fit of the ZM law.

In what follows, observations are assumed to be conditionally independent.
Thus, the log-likelihood satisfies the linear form restriction and the ml model lf can be used to maximize.

Following Baixeries et al. (2013), the likelihood function for one single observation with rank r and the corresponding frequency f_r can be defined as:

l_r=p_r^(f_r)

Taking logs on both sides and using the definitions presented above,
this yields the log likelihood function:

ll_r=-a*f_r*log(r+ß)-f_r*log(H(N,a,ß)).

Saved results

All results saved after ml, additionally zipffit saves the parameter of the Zipf distribution (e(zalpha))
or the parameters of the Zipf-Mandelbrot distribution (e(zmalpha),e(zmbeta)). In both cases, the number of different observations
(e(types)) and the sum of the var frequencies (e(tokens)) are saved.

Examples:

Generate test data (taken from Izsák, 2006).

. clear

. set obs 18

. gen test=1

. replace test=145 in 1

. replace test=96 in 2

. replace test=35 in 3

. replace test=29 in 4

. replace test=20 in 5

. replace test=11 in 6

. replace test=4 in 7/9

. replace test=3 in 10/11

. replace test=2 in 12/13

ML fit of the Zipf distribution.

. zipffit test

ML fit of the Zipf-Mandelbrot distribution with maximize_options.

. zipffit test, zmand iterate(100) difficult

ML fit of the Zipf-Mandelbrot distribution with the iteration log displayed.

. zipffit test, zmand log difficult

ML fit of the Zipf-Mandelbrot distribution for the first 10 ranks.

. zipffit test, zmand rank(10)

Author

Alexander Koplenig <koplenig@ids-mannheim.de>, Institute for the
German Language (IDS), Mannheim, Germany.

References

For this help file and for some code lines in the
actual ado file, the paretofit.ado written by
Stephen P. Jenkins & Philippe Van Kerm was used as a blueprint (cf. paretofit).

Baixeries, J., Elvevåg, B., & Ferrer-i-Cancho, R. (2013).
The Evolution of the Exponent of Zipf’s Law in Language Ontogeny.
PLoS ONE, 8(3), e53227. doi:10.1371/journal.pone.0053227

Izsák, F. (2006). Maximum likelihood estimation for constrained parameters of multinomial distributions—Application
to Zipf–Mandelbrot models. Computational Statistics & Data Analysis, 51(3), 1575–1583.

Mandelbrot, B. (1953). An informational theory of the statistical structure of language.
In W. Jackson (Ed.), Communication Theory (pp. 468–502)
London: Butterworths Scientific Publications.

Zipf, G. K. (1935). The psycho-biology of language?;
an introduction to dynamic philology.
Boston: Houghton Mifflin company.