help for jaccard

Jaccard similarity or dissimilarity of sets

jaccard
varlist
[weight]
[if exp]
[in range]

[
,
{
complement
|
count
}
upper
lower
diagonal
varlabels
graph_options
savedata(filespec)
]

fweights and aweights may be specified.

Description

jaccard calculates the Jaccard measure of similarity of two or
more sets, or its complement, and plots results in a tabular bar chart.
Set membership is specified by a bundle of indicator variables.
Optionally, the count in each intersection may be plotted instead.

For sets A, B the number of elements in their intersection
|A intersection B| is divided by the number of elements in
their union |A union B| to give a measure between 0 (A
and B are disjoint) and 1 (A and B are identical). Here
|.| indicating number of elements may be a count or some other measure
of abundance.

Note: SMCL supports display on graphs of the standard cap and cup symbols for
intersection and union, but they will not show up in help files. See
text.

The order of variables presented to jaccard affects the order of
sets in the plot and in the associated dataset.

Commonly, but not necessarily, subset frequencies (or abundances) are
already in a variable in the dataset and so that variable should be
specified as frequency or analytic weights. If no weights are specified,
jaccard counts observations for you. Either way, note that the
focus of this command is on displaying similarity or dissimilarity of
sets, and not the particular observations in each subset.

The display uses tabplot (Cox 2016b, 2017, 2020, 2022), which
must be installed separately. Each row and column are labelled by
variable names or optionally by variable labels. In either case,
variable names or variable labels, short text is desirable.

The code uses labmask (Cox 2008), which must be installed
separately.

The reduced dataset used by jaccard may be saved for future work
using the savedata() option. This dataset may be as useful as or
more useful than the plot. Saving results allows greater flexibility in
plotting. Tabulation or other reporting is also made easier.

The variables in such a reduced dataset are as follows.

_name1 and _name2 are string variables containing the
original variable names or variable labels.

_which1 and _which2 are numeric variables with value labels
from the original variable names or variable labels.

_inter is a numeric variable measuring the frequency or abundance
of each intersection.

_union is a numeric variable measuring the frequency or abundance
of each union.

_Jaccard is a numeric variable containing the Jaccard measure or
its complement.

Allenby and Slomson (2011, p.14) comment: "There is, unfortunately, no
standard notation for the number of elements in a set". They could have
added "and no standard term either". Terms encountered (other than
"number of elements") include cardinality, order, potency, power, and
the homely size. See annotations of several references.

More detailed Remarks, including many further references, follow later in
this help.

Options

complement specifies calculating 1 minus the usual Jaccard
measure, as a measure of dissimilarity.

count specifies plotting the intersection counts (more generally,
abundances) rather than
the Jaccard measure or its complement.

complement and count may not be combined.

upper specifies display of the upper half of the bar matrix.

lower specifies display of the lower half of the bar matrix.

diagonal specifies display of the diagonal of the bar matrix.

The three options above may be combined. The default is
lower.

varlabels specifies use of variable labels to describe each set.
The default is to use variable names. If a variable label has not been
defined, the variable name is used instead.

graph_options are other options of tabplot. Defaults
include xtitle("") ytitle("") aspect(1).

savedata() specifies a (filepath and) filename for saving results
to a new dataset. The specification may include , replace -
which is needed to replace any existing dataset with the same path and
name.

Saved results

The tabplot command used is saved in local macro graph_cmd
in the program's space.

Examples

. local opts showval(format(%5.4f)) lcolor(blue) fcolor(blue*0.3)

. set more off

. set scheme s1color

. * EXAMPLE 1

. * Schnable et al. 2009 counts of gene families

. clear

. input Rice Maize Sorghum Arabidopsis freq

1 0 0 0 1110

1 1 0 0 229

0 1 0 0 465

1 0 1 0 661

1 1 1 0 2077

0 1 1 0 405

0 0 1 0 265

1 0 1 1 304

1 1 1 1 8494

0 1 1 1 112

0 0 1 1 34

1 0 0 1 81

1 1 0 1 96

0 1 0 1 11

0 0 0 1 1058

end

. label var Arabidopsis "{it:Arabidopsis}"

. jaccard A R M S [fw=freq], `opts' varlabels frame(1) name(JC1, replace)

. jaccard A R M S [fw=freq], `opts' varlabels lower diagonal frame(1) name(JC2, replace)

. jaccard A R M S [fw=freq], `opts' varlabels lower diagonal complement frame(1) name(JC3, replace)

. * EXAMPLE 2

. * D'Hont et al. 2012

. clear

. input byte(Phoenix Musa Brachypodium Sorghum Oryza Arabidopsis) float freq str52 name

1 1 1 1 1 1 7674 "Phoenix Musa Brachypodium Sorghum Oryza Arabidopsis"

1 1 1 1 1 0 685 "Phoenix Musa Brachypodium Sorghum Oryza"

1 1 1 1 0 1 113 "Phoenix Musa Brachypodium Sorghum Arabidopsis"

1 1 1 1 0 0 24 "Phoenix Musa Brachypodium Sorghum"

1 1 1 0 1 1 80 "Phoenix Musa Brachypodium Oryza Arabidopsis"

1 1 1 0 1 0 18 "Phoenix Musa Brachypodium Oryza"

1 1 1 0 0 1 7 "Phoenix Musa Brachypodium Arabidopsis"

1 1 1 0 0 0 12 "Phoenix Musa Brachypodium"

1 1 0 1 1 1 149 "Phoenix Musa Sorghum Oryza Arabidopsis"

1 1 0 1 1 0 62 "Phoenix Musa Sorghum Oryza"

1 1 0 1 0 1 23 "Phoenix Musa Sorghum Arabidopsis"

1 1 0 1 0 0 19 "Phoenix Musa Sorghum"

1 1 0 0 1 1 28 "Phoenix Musa Oryza Arabidopsis"

1 1 0 0 1 0 35 "Phoenix Musa Oryza"

1 1 0 0 0 1 206 "Phoenix Musa Arabidopsis"

1 1 0 0 0 0 467 "Phoenix Musa"

1 0 1 1 1 1 258 "Phoenix Brachypodium Sorghum Oryza Arabidopsis"

1 0 1 1 1 0 190 "Phoenix Brachypodium Sorghum Oryza"

1 0 1 1 0 1 11 "Phoenix Brachypodium Sorghum Arabidopsis"

1 0 1 1 0 0 23 "Phoenix Brachypodium Sorghum"

1 0 1 0 1 1 5 "Phoenix Brachypodium Oryza Arabidopsis"

1 0 1 0 1 0 12 "Phoenix Brachypodium Oryza"

1 0 1 0 0 1 3 "Phoenix Brachypodium Arabidopsis"

1 0 1 0 0 0 25 "Phoenix Brachypodium"

1 0 0 1 1 1 21 "Phoenix Sorghum Oryza Arabidopsis"

1 0 0 1 1 0 42 "Phoenix Sorghum Oryza"

1 0 0 1 0 1 4 "Phoenix Sorghum Arabidopsis"

1 0 0 1 0 0 49 "Phoenix Sorghum"

1 0 0 0 1 1 6 "Phoenix Oryza Arabidopsis"

1 0 0 0 1 0 32 "Phoenix Oryza"

1 0 0 0 0 1 105 "Phoenix Arabidopsis"

1 0 0 0 0 0 769 "Phoenix"

0 1 1 1 1 1 1458 "Musa Brachypodium Sorghum Oryza Arabidopsis"

0 1 1 1 1 0 368 "Musa Brachypodium Sorghum Oryza"

0 1 1 1 0 1 54 "Musa Brachypodium Sorghum Arabidopsis"

0 1 1 1 0 0 13 "Musa Brachypodium Sorghum"

0 1 1 0 1 1 29 "Musa Brachypodium Oryza Arabidopsis"

0 1 1 0 1 0 28 "Musa Brachypodium Oryza"

0 1 1 0 0 1 7 "Musa Brachypodium Arabidopsis"

0 1 1 0 0 0 9 "Musa Brachypodium"

0 1 0 1 1 1 71 "Musa Sorghum Oryza Arabidopsis"

0 1 0 1 1 0 64 "Musa Sorghum Oryza"

0 1 0 1 0 1 21 "Musa Sorghum Arabidopsis"

0 1 0 1 0 0 49 "Musa Sorghum"

0 1 0 0 1 1 13 "Musa Oryza Arabidopsis"

0 1 0 0 1 0 29 "Musa Oryza"

0 1 0 0 0 1 155 "Musa Arabidopsis"

0 1 0 0 0 0 759 "Musa"

0 0 1 1 1 1 206 "Brachypodium Sorghum Oryza Arabidopsis"

0 0 1 1 1 0 2809 "Brachypodium Sorghum Oryza"

0 0 1 1 0 1 14 "Brachypodium Sorghum Arabidopsis"

0 0 1 1 0 0 402 "Brachypodium Sorghum"

0 0 1 0 1 1 18 "Brachypodium Oryza Arabidopsis"

0 0 1 0 1 0 547 "Brachypodium Oryza"

0 0 1 0 0 1 10 "Brachypodium Arabidopsis"

0 0 1 0 0 0 387 "Brachypodium"

0 0 0 1 1 1 40 "Sorghum Oryza Arabidopsis"

0 0 0 1 1 0 1151 "Sorghum Oryza"

0 0 0 1 0 1 9 "Sorghum Arabidopsis"

0 0 0 1 0 0 827 "Sorghum"

0 0 0 0 1 1 6 "Oryza Arabidopsis"

0 0 0 0 1 0 1246 "Oryza"

0 0 0 0 0 1 1187 "Arabidopsis"

0 0 0 0 0 0 . ""

end

. jaccard P-A [w=freq], `opts' frame(1) name(JC4, replace) xla(, labsize(small)) yla(, labsize(small))

. * EXAMPLE 3

. * incidence of missing values in nlswork.dta

. webuse nlswork, clear

. * missings from Stata Journal: search dm0085, entry

. capture noisily missings report

. foreach v in ind_code union wks_ue tenure wks_work {

. gen M`v' = missing(`v')

. label var M`v' "`v'"

. }

. jaccard M*, `opts' varlabels name(JC5, replace)

. jaccard M*, `opts' varlabels complement name(JC6, replace)

. * EXAMPLE 4

. * various indicators in nlswork.dta

. webuse nlswork, clear

. jaccard nev_mar c_city collgrad south, `opts' name(JC7, replace)

. label var nev_mar "never married"

. label var c_city "central city"

. label var collgrad "college graduate"

. label var south "South"

. jaccard nev_mar c_city collgrad south, `opts' varlabels name(JC8, replace)

Author

Nicholas J. Cox, University of Durham

n.j.cox@durham.ac.uk

Acknowledgment

Tim Morris made several helpful suggestions.

Also see

Help for

groups (Stata Journal) (if installed)

upsetplot (if installed)

vennbar (if installed)

Remarks

jaccard requires a bundle of numeric variables with values 0 or 1.
Such variables are variously called indicator, dummy, binary,
dichotomous, zero-one, Boolean, logical or quantal. Values other than 0
or 1, including missing values, will be ignored. Observations used will
thus have all values 0 or 1 in all variables specified. Otherwise put,
jaccard is not for arbitrary categorical variables which may have
3 or more distinct values, except in so far as they are represented by a
set of such indicator variables.

If your dataset is already aggregated to frequencies or other measures
of abundance, specify those as weights multiplying the indicator
variables.

The elementary but fundamental idea of representing true (or present) as
1 and false (or absent) as 0 has a splendid history. Although it has
yet longer roots, the idea was strongly developed by George Boole
(1815-1864): Boole (1854) was his major work in this territory, on
which see particularly Grattan-Guinness (2005). Boole has been given a
full-length biography (MacHale 2014) and an even longer sequel (MacHale
and Cohen 2018). For shorter accounts see Gardner (1969), Broadbent
(1970), MacHale (2000, 2008), Heath and Seneta (2001), or
Grattan-Guinness (2004). See (e.g.) Dewdney (1993) or Gregg (1998) for
samples of how such Boolean algebra features in computing. See Knuth
(1998, Ch.4.1) for an excellent historical summary of positional number
systems and Knuth (2011) for a masterly synopsis, including historical
material, of related combinatorial algorithms. See Strickland and Lewis
(2022) for a focus on binary arithmetic and logic in the work of Leibniz
(1646-1716). The leading biography of Leibniz is by Antognazza
(2009), although the earlier biography by Aiton (1985) is still very
helpful. Leibniz's projects feature in many sub-plots in Stephenson
(2003) and its sequels. Cox (2016a) makes further Stata-related
comments on truth, falsity, and indication. Cox and Schechter (2019)
survey the creation of indicator variables in Stata.

Friendly introductions to set theory include Stewart (1975) and Gullberg
(1997). Conversely, compare Hamming (1985, p.367): "Set theory has been
taught until the typical student is weary of it, so we will assume that
it is familiar." Historically and to the present set theory is linked to
much fundamental work in logic, number theory, and other parts of
mathematics (Bagaria 2008; various chapters in Grattan-Guinness 1994;
Stillwell 2010).

For more on Jaccard and the Jaccard measure, see [MV] measure_option and
its references. There is no immediate ambition to extend this command to
other measures in this territory.

References

Aiton, E.J. 1985.
Leibniz: A Biography.
Bristol: Adam Hilger.

Allenby, R.B.J.T. and A. Slomson. 2011.
How to Count: An Introduction to Combinatorics.
Boca Raton, FL: CRC Press.

Antognazza, M.R. 2009.
Leibniz: An Intellectual Biography.
New York: Cambridge University Press.

Bagaria, J. 2008.
Set theory.
In Gowers, T. (ed.)
The Princeton Companion to Mathematics.
Princeton, NJ: Princeton University Press, 615-634.

Biggs, N.L. 2002.
Discrete Mathematics.
Oxford: Oxford University Press.
p.48 size, cardinality

Boole, G. 1854.
An Investigation of the Laws of Thought on Which are Founded the Mathematical Theories of Logic and Probabilities.
London: Walton and Maberley; Cambridge: Macmillan.

Broadbent, T.A.A. 1970.
Boole, George.
In Gillispie, C.C. (ed.} Dictionary of Scientific Biography.
New York: Charles Scribner's Sons 2: 293-298.

Cameron, P.J. 1994.
Combinatorics: Topics, Techniques, Algorithms.
Cambridge: Cambridge University Press.
p.16 cardinality

Cox, N.J. 2008.
Between tables and graphs.
Stata Journal 8: 269-289.

Cox, N.J. 2016a.
Truth, falsity, indication, and negation.
Stata Journal 16: 229-236.

Cox, N.J. 2016b.
Multiple bar charts in table form.
Stata Journal 16: 491-510.

Cox, N.J. 2017.
Multiple bar charts in table form.
Stata Journal 17: 779.

Cox, N.J. 2020.
Multiple bar charts in table form.
Stata Journal 20: 757.

Cox, N.J. 2022.
Multiple bar charts in table form.
Stata Journal 22: 467.

Cox, N.J. and C.B. Schechter. 2019.
How best to generate indicator or dummy variables.
Stata Journal 19: 246-259.

Crossley, J.N., J. Ash, C.J. Brickhill, J.C. Stillwell and N.H.
Williams. 1972.
What is Mathematical Logic?
London: Oxford University Press.
p.69 cardinality

Dewdney, A.K. 1993.
The New Turing Omnibus: 66 Excursions in Computer Science.
New York: Henry Holt.

D'Hont, A. and many authors. 2012.
The banana (Musa acuminata) genome and the evolution of monocotyledonous plants.
Nature 488: 213-217. https://doi.org/10.1038/nature11241

Gardner, M. 1969.
Boolean algebra, Venn diagrams and the propositional calculus.
Scientific American 220(2): 110-114.
https://www.jstor.org/stable/pdf/24926287.pdf
Reprinted as Boolean algebra in 1979.
Mathematical Circus. New York: Alfred A. Knopf, 87-101.

Graham, R.L., D.E. Knuth and O. Patashnik. 1994.
Concrete Mathematics: A Foundation for Computer Science.
Reading, MA: Addison-Wesley.
p.xi cardinality

Grattan-Guinness, I. (ed.) 1994.
Companion Encyclopedia of the History and Philosophy of the Mathematical Sciences.
London: Routledge.

Grattan-Guinness, I. 2004.
Boole, George.
In Matthew, H.C.G. and B. Harrison (eds)
Oxford Dictionary of National Biography.
Oxford: Oxford University Press 6: 582-585.

Grattan-Guinness, I. 2005.
George Boole,
An Investigation of the Laws of Thought on Which are Founded the Mathematical Theories of Logic and Probabilities (1854).
In Grattan-Guinness, I. (ed.)
Landmark Writings in Western Mathematics 1640-1940.
Amsterdam: Elsevier, 470-479.

Green, J.A. 1965.
Sets and Groups.
London: Routledge and Kegan Paul.
p.2 order [of a finite set]

Green, J.A. 1988.
Sets and Groups: A First Course in Algebra.
London: Chapman & Hall.
p.2 order [of a finite set]

Gregg, J.R. 1998.
Ones and Zeros: Understanding Boolean Algebra, Digital Circuits, and the Logic of Sets.
Piscataway, NJ: IEEE Press.

Grimmett, G.R. and D.R. Stirzaker. 2020.
Probability and Random Processes.
Oxford: Oxford University Press.
p.6 cardinality

Gullberg, J. 1997.
Mathematics: From the Birth of Numbers.
New York: W.W. Norton.

Hamming, R.W. 1985.
Methods of Mathematics Applied to Calculus, Probability, and Statistics.
Englewood Cliffs, NJ: Prentice-Hall.

Heath, P. and E. Seneta. 2001.
George Boole.
In Heyde, C.C. and E. Seneta (eds)
Statisticians of the Centuries.
New York: Springer, 167-170.

Ito, K. (ed.) 1987.
Encyclopedic Dictionary of Mathematics.
Cambridge, MA: MIT Press. [macron accent on o of Ito]
potency, power

James, G. and R.C. James. 1992.
Mathematics Dictionary.
New York: Van Nostrand Reinhold.
potency, power

Knuth, D.E. 1997.
The Art of Computer Programming: Volume 1: Fundamental Algorithms.
Reading, MA: Addison-Wesley.
p.625 cardinality

Knuth, D.E. 1998.
The Art of Computer Programming: Volume 2: Seminumerical Algorithms.
Reading, MA: Addison-Wesley.

Knuth, D.E. 2011.
The Art of Computer Programming: Volume 4A: Combinatorial Algorithms, Part 1.
Upper Saddle River, NJ: Addison-Wesley.

Liebeck, M. 2016.
A Concise Introduction to Pure Mathematics.
Boca Raton, FL: CRC Press.
p.195 cardinality

MacHale, D. 2000.
George Boole 1815-1864.
In Houston, K. (ed.)
Creators of Mathematics: The Irish Connection.
Dublin: University College Dublin Press, 27-32.

MacHale, D. 2008.
George Boole.
In Gowers, T. (ed.)
The Princeton Companion to Mathematics.
Princeton, NJ: Princeton University Press, 769-770.

MacHale, D. 2014.
The Life and Work of George Boole: A Prelude to the Digital Age.
Cork: Cork University Press.

MacHale, D. and Y. Cohen. 2018.
New Light on George Boole.
Cork: Cork University Press.

Schnable, P.S. and many co-authors. 2009.
The B73 maize genome: complexity, diversity, and dynamics.
Science 326: 1112-1115.
http://www.jstor.org/stable/27736489

Stephenson, N. 2003.
Quicksilver: Volume One of the Baroque Cycle.
New York: William Morrow.

Stewart, I. 1975.
Concepts of Modern Mathematics.
Harmondsworth: Penguin.

Stillwell, J. 2010.
Mathematics and Its History.
New York: Springer.

Strickland, L. and H.R. Lewis. 2022.
Leibniz on Binary: The Invention of Computer Arithmetic.
Cambridge, MA: MIT Press.

Zeitz, P. 2007.
The Art and Craft of Problem Solving.
Hoboken, NJ: John Wiley.
p.147 cardinality