*! jl 1.0.0 8apr2024
jl

Title

Bridge to Julia

Syntax

jl: juliaexpr

_jl: juliaexpr

where juliaexpr is an expression to be evaluated in Julia.

jl subcommand [varlist], [options]

subcommand

start Start Julia, optionally setting number of threads

use Load Julia DataFrame to current Stata data set

save Save current data set to Julia DataFrame

PutVarsToDF Copy Stata variables to Julia DataFrame

GetVarsFromDF Copy Stata variables from Julia DataFrame, mapping NaN to missing

PutVarsToMat Copy Stata variables to Julia matrix

GetVarsFromMat Copy Stata variables from Julia matrix, mapping NaN to missing

PutMatToMat Copy Stata matrix to Julia matrix, mapping missing to NaN

GetMatFromMat Copy Stata matrix from Julia matrix, mapping NaN to missing

SetEnv Switch to named package environment

GetEnv Get name of current package environment

AddPkg Install Julia package if not installed, or update if version below threshold

version Get version of installed jl package

jl start [, threads(integer or auto)]

jl use dataframename, [clear]

-or-

jl use varlist using dataframename, [clear]

jl save [dataframename], [nolabel nomissing doubleonly]

jl PutVarsToDF [varlist] [if] [in], [destination(string) cols(string) nolabel nomissing doubleonly]

jl GetVarsFromDF varlist [if] [in], [cols(string) source(string) replace nomissing]

jl PutVarsToMat [varlist] [if] [in], destination(string)

jl GetVarsFromMat varlist [if] [in], source(string) [replace]

jl PutMatToMat matname, [destination(string)]

jl GetMatFromMat matname, [source(string)]

jl SetEnv [name]

jl GetEnv

jl AddPkg name, [minver(string)]

jl version

Description

jl gives access from the Stata prompt to the free programming language Julia. It provides three
sets of tools:

1. The jl: prefix command, which allows you to send commands to Julia and see the results. (But results
are suppressed for lines ending with ";".) Example:
jl: "Hellow world!". Typing jl: or jl by itself starts an interactive mode, in which
multiple lines can be typed. This mode stops when the user types exit().

2. Subcommands, listed above, for high-speed copying of data between Julia and Stata, managing installation of Julia packages.

3. A library of Julia functions to allow reading and writing of Stata variables, macros, matrices, and scalars. Most of
the functions hew closely to those in the Stata plugin interface. For example,
jl: SF_vdata(1, 2) getsts the value in the second row of the first variable in the Stata data set. Some
are higher level: st_data("price mpg") returns a two-column matrix with the contents of those variables.

Because Julia does just-in-time-compilation, Julia-based commands take longer on first use in a Stata session and even longer on first use on a given machine.

The interactive mode does not work in do files. That is, while you can begin a Mata or Python
block in a do file withe the mata or python command, you cannot do the same for
Julia. You can work around that limitation. You can put several commands in one
line, separating them with semicolons. And you can break what are logically single lines into many, using Stata's
continuation token, "///":

jl: s = 0; for i in 1:10 s += i end; s

jl: s = 0; ///

for i in 1:10 ///

s += i ///

end; ///

s

The jl start command is often not needed. If it is not used, then Julia will be automatically started anyway the first time it is called in a Stata
session. The one reason to call jl start is to control the number of CPU threads available to Julia, through the threads() option. Many Julia
programs exploit multithreading for speed. In Windows and Linux, but not macOS, you can also control the number of threads by editing the JULIA_NUM_THREADS
environment variable. jl start will have no effect unless it comes before every other jl command in a Stata session, and before any use of
packages such as reghdfejl and boottest that call jl. For more, see section on threads below. You
do not need Stata/MP to run mulithreaded Julia code.

The data-copying subcommands come in high- and low-level variants. The high-level jl use and jl save subcommands have similar syntax
to Stata's use and save, but copy to and from Julia DataFrames. Unlike the low-level jl GetVarsFromDF, jl use
will clear the current data set if the clear option is included, and ensure that the new data set has enough rows to receive all the data.

The low-level routines give more include options to improve performance that are
useful when using jl to write a Julia back end for a Stata package. By default, the jl PutVarsToDF subcommand will map Stata data columns
to Julia DataFrame columns of corresponding type, and mark all destination columns to allow missing values. jl GetVarsFromDF does
something similar in the other direction. PutVarsToDF's nomissing option increases speed and is appropriate for variables known to contain no missing values.
Another time-saving option, doubleonly, causes jl PutVarsToDF to map all numeric Stata data types, to double-precision numbers--called double in Stata
and Float64 in Julia. Treatment of strings is then undefined. Without this option, the target
columns will have the same types as the source columns.

When copying to Julia, any existing DataFrame or matrix of the same name is overwritten.

The subcommands mapping between Stata data and Julia DataFrames translate between Stata variables with value labels and Julia categorical vectors. However,
the mapping from Stata to Julia is computationally inefficient--all values are replaced with their labels before conversion to type CategoricalVector, and
can be prevented with the nolabel option.

The SetEnv subcommand switches to a package environment associated with the supplied
name. This is useful when writing Julia-based Stata programs that need to install certain Julia
packages. Switching to a dedicated environment minimizes version conflicts with packages
downloaded for other purposes. The directory used for the dedicated environment will be a
subdirectory of Julia's default package environment directory, for example,
"`~/.julia/environments/v1.10/MyEnvironment". It will be created if it does not exist. If new,
it will only have the DataFrames package. Calling SetEnv without any arguments reverts to
the default package environment.

The GetEnv displays the name and location of the current package environment and
saves the results as r() macros.

The AddPkg subcommand updates a package to the latest version in Julia's general registry if the package is not installed at all, or if
the current version is below that set by the optional minver() option. It operates within
the current Julia package environment. So jl AddPkg ... followed by jl SetEnv ... will generally
have a different effect than the same commands in the opposite order.

Installing Julia

This package is designed for 64-bit Windows, Linux, and macOS, the last on an Intel or Apple CPU. It requires
Julia 1.9.4 or higher. As documented here, the easiest way to
install it in Windows is from the Microsoft Store; and the
easiest way to install it in Linux and macOS is with this shell command from inside Stata, which is clickable:

! curl -fsSL https://install.julialang.org | sh -s -- -y

On Intel Macs,
jl seems to require at least macOS 11 (Big Sur) or 12 (Monterey). On computers not officially supported by those editions, one can use
the OpenCore Legacy Patcher to upgrade--at your own risk.

After installing Julia, restart Stata for good measure.

On first use, jl will attempt to install the Julia packages DataFrames.jl and CategoricalArrays.jl. That can take a minute.

Installing the JuliaMono font

The JuliaMono font is a free, monospaced font that works well in Stata and includes
Unicode characters that often appear in Julia output, such as vertical and
diagonal ellipses. You can
download the TTF file, install it, and
configure Stata to use it.

Setting the number of CPU threads

Many Julia programs exploit multithreading. However, Julia is typically installed to only allow one thread by default. In Windows and Linux, and in
macOS if you willing to endure a little inconvenience (see below), you can
permanently override this default by editing your operating system's JULIA_NUM_THREADS environment variable. Set it to an integer such as 4 or 8,
or to "auto" to let Julia decide. On CPUs with hyperthreading or efficiency (E) cores as well as performance (P) cores,
the optimal number is usually not the maximum the CPU technically supports. A good guess is the number of P cores.

How to set this variable depends on your operating system:

* In Windows, use the Environment Variables control panel to add JULIA_NUM_THREADS. One route to that dialog box is to press the Windows logo
button on the keyboard and type "environment variables".

* In Linux, add "export JULIA_NUM_THREADS=8" (as an example) to the text file "~/.bashrc". Restart the terminal window.

* Similarly, in macOS, add such a line to "~/.zshenv".

Now, in macOS, editing ~/.zshenv will have no effect unless you launch Stata from a terminal window instead of the graphical desktop. To do so:

* Type Command-spacebar to launch Spotlight Search, then type "Stata" and observe the autocompletion to determine the exact name of the Stata app on your
system. For example, if you have Stata/MP, you should see "StataMP".

* Clear the Spotlight Search box and then type "terminal" and the return key to launch a terminal window.

* Run the command line "open -a [Stata app name]" where [Stata app name] is, for example, "StataMP".

An alternative way to set the number of threads is to explicitly launch Julia inside Stata with jl start, threads(string) where string is a positive
integer or auto. To have any effect, this command must precede any other invocation of jl in a Stata session,
and any other invocation of Julia-calling commands such as reghdfejl and boottest.

To determine how many threads are available, type "jl: Threads.nthreads()" at the Stata prompt.

Options

In the data-copying subcommands, the varlist's and matname's before the commas always
refer to Stata variables or matrices. If a varlist is omitted where it is optional,
the variable list will default to *, i.e., all variables in the current data frame in
their current order.

The options after the comma in these subcommands refer to Julia objects. destination()
and source() name the Julia matrix or DataFrame to be written to or from. When
a DataFrame name is not provided, it defaults to df. The cols() option specifies the
DataFrame columns to be copied to or from. It defaults to the Stata varlist before the comma.

Destination Stata matrices and Julia matrices and DataFrames are entirely replaced. Destination Stata
variables will be created or, if replace is specified, overwritten, subject to any
[if] [in] restriction.

The _jl: prefix command is for programmers. It works the same as jl:, except that for the sake of speed it disables
certain features of jl: that enhance the interactive Julia experience within Stata. These include checking whether Julia is
started and starting it if not, showing output from print()
and other commands, not generating syntax errors partway through multi-line code blocks such as for loops, and
interpreting soft-scoped assignments as if in interactive mode. The
time savings from _jl: can be small in absolute terms (~0.01 seconds per call). But it adds up in a program that issues many
Julia commands. Warning: _jl can crash Stata if it is called before Julia is started via a jl command!

Stored results

jl: stores the output in the macro r(ans). jl version returns the version number of the installed jl package in r(version).

Stata interface functions

The julia.ado package automatically loads a Julia module that gives access to the
Stata Plugin Interface (SPI), which see for more information on
syntax. The functions in the module allow one to read and write
Stata objects from Julia. They can be roughly divided into low- and high-level groups. The low-level
functions closely mimic the foundational SPI functions, which let you, for example, determine
the size of the data set and read and write individual data points. The high-level functions
work similarly to the Mata functions they are named after: st_global(), st_local(), st_matrix(), st_data(), and
st_view(). In particular, st_view() lets you treat one or more Stata variables as a matrix, which can be read
and written with Julia commands.

Function

SF_nobs() Number of observations in Stata data set

SF_nvar() Number of variables

SF_var_is_string(i::Int) Whether variable i is string

SF_var_is_strl(i::Int) Whether variable i is a strL

SF_var_is_binary(i::Int, j::Int) Whether observation i of variable j is a binary strL

SF_sdatalen(j::Int, i::Int) String length of variable i, observation j

SF_is_missing() Whether a Float64 value is Stata missing

SV_missval() Stata floating-point value for missing

SF_vstore(j::Int, i::Int, val::Real) Set observation j of variable i to val (numeric)

SF_sstore(j::Int, i::Int, s::String) Set observation j of variable i to s (string)

SF_vdata(j::Int, i::Int) Get observation j of variable i (numeric)

SF_sdata(j::Int, i::Int) Get observation j of variable i (string)

SF_macro_save(mac::String, tosave::String) Set macro value

SF_macro_use(mac::String) Get macro mac

SF_scal_save(scal::String, val::Real) Set scalar value

SF_scal_use(scal::String) Get scalar scal

SF_row(mat::String) Number of rows of matrix mat

SF_col(mat::String) Number of columns of matrix mat

SF_mat_store(mat::String, i::Int, j::Int, val::Real) mat[i,j] = val

SF_mat_el(mat::String, i::Int, j::Int) Get mat[i,j]

SF_display(s::String) Print to Stata results window

SF_error(s::String) Print error to Stata results window

st_nobs() Number of observations in Stata data set; same as SF_nobs()

st_nvar() Number of Stata variables; same as SF_nvar()

st_varindex(s::String) Index in data set of variable named s

st_global(mac::String) Get global macro mac

st_global(mac::String, tosave::String) Set global macro mac

st_local(mac::String, tosave::String) Set local macro mac

st_numscalar(scal::String) Get scalar scal; same as SF_scal_use()

st_numscalar(scal::String, val::Real) Set scalar scal; same as SF_scal_save()

st_matrix(matname::String) Get numeric Stata matrix

st_matrix(matname::String, jlmat::Matrix) Put Julia matrix in pre-existing Stata matrix

st_data(varnames::String) Get Stata variables in space-delimited list, as matrix

st_data(varnames::Vector{<:String}) Get Stata variables in string vector, as matrix

st_data(varnames::String, sample::Vector{Bool}) Get Stata variables, with sample restriction, as matrix

st_data(varnames::Vector{<:String}, sample::Vector{Bool}) Get Stata variables, with sample restriction, as matrix

st_view(varnames::String) Get Stata variables in space-delimited list, as view

st_view(varnames::Vector{<:String}) Get Stata variables in string vector, as view

st_view(varnames::String, sample::Vector{Bool}) Get Stata variables, with sample restriction, as view

st_view(varnames::Vector{<:String}, sample::Vector{Bool}) Get Stata variables, with sample restriction, as view

Used with jl, but not _jl, st_local() allows one to write locals. One-line jl: and _jl: commands
can access locals by quoting them, such as with jl: X = st_data("`varnames'")

Author

David Roodman

david@davidroodman.com

Acknowledgements

This project was inspired by James Fiedler's Python plugin for Stata (as perhaps
was Stata's support for Python).