*! version 2.8 15 Dec 2022

Title

inshell -- An enhanced multi-platform shell wrapper that will send commands to the operating system's shell while capturing and printing their standard output, standard error, and native shell return codes.

Syntax

inshell operating_system_command [cmd_args] (1)

inshell /absolute/path/to/program [program_args] (2)

inshell . /path/to/script/file [script_args] (3)

inshell {cd|chdir} [/directory/to/change/to] (4)

Description

inshell is a shell wrapper that is designed to enhance both the interactive and the scripted use of the shell in Stata.
It can execute commands or scripts via the operating system's shell and will display the standard output in the Results window while capturing it into a series of local macros.
In the event of an error, it will also capture and display both the standard error and the shell's native return code.
Additionally, it contains an enhanced wrapper for cd. It runs on macOS, Linux, and Microsoft Windows Command shells.
It also works with Microsoft PowerShell on all operating systems, with some limitations.

Remarks

This help file and its examples are geared towards users of operating systems with Unix shells (macOS and Linux), but the same basic principles apply to those that use Microsoft Windows.
These remarks also serve to detail advanced usage of Stata's existing shell command.

Notation

It is important to note that commands sent directly to inshell, whether entered into the Command window or run as part of a do-file, will pass through Stata's command pre-processor before they are executed.

Because shell code uses the same dollar sign alphanumeric notation (and often the same bracing structure) for variable referencing that Stata uses for its own global macros,
and because all commands must pass through this pre-processor before they are executed, Stata will attempt to perform macro substitution upon each and every substring that could be a valid name for a Stata global.
This will happen regardless of whether commands are sent via shell, inshell or any other package.
Therefore if your code is intended to reference a variable within the shell's environment you will need to escape expressions of the form
$var1 or ${var2} to \$var1 or \${var2}. For instance, to print the $PATH on macOS and Linux:

. inshell echo \$PATH (click to run)

In this case it is necessary to use the single backslash (\) to escape the dollar sign ($) otherwise Stata will look for a global macro named PATH.
If it does not exist, Stata will replace that part of the command with a zero-length string because the macro substitution will result in an empty value being passed to what Stata interprets as its own global macro.

However, any dollar signs ($) used for command substitution or in another setting not directly followed by a letter or an underscore (_) need not be escaped. To use command substitution:

. inshell echo $(date +%s) >> \${LOG_DIR}/logfile_$(date +%m%d%Y).log

or

. inshell echo `date +%s` >> \${LOG_DIR}/logfile_`date +%m%d%Y`.log

Common shell parameters such as $0 (the current shell), $$ (the current shell's process ID), or $? (the previous command's return code) also need not be escaped.

local macros also need not be escaped as they have a different structure -- formed between a back-tick (`) and a single apostrophe (')
-- than double back-tick command substitution, which is performed between two single back-ticks: (`) command (`).

To use a Stata local or global macro, leave it un-escaped:

. global id a5b7d9e4f1

. local datetime $S_DATE $S_TIME

. inshell echo id: $id >> \${LOG_DIR}/results_${id}_`date +%m%d%Y`.log

. inshell echo `datetime' >> \${LOG_DIR}/results_${id}_`date +%m%d%Y`.log

Note that in the example above, braces must be used around the global macro id as the underscore immediately following it would otherwise be included by Stata's macro parsing routine.

Windows Command shell uses double percent sign (%) notation which does not require escaping:

. inshell echo %PATH% (Windows Command shell)

As of the date of this printing, there are no other publicly available shell wrappers for Stata for Mac, Stata for Linux,
or for users of Microsoft PowerShell running on a Mac or Linux system that are able to access dollar sign alphanumeric notation shell variables.

Environment

Sourcing of shell configuration files to produce the shell's environment will differ across each Stata installation for several obvious reasons, however,
terminal-launched instances (also called console instances) of Stata will assemble their environments from a broader range of sources than those used by the GUI application.
This is because the Stata Results window in the GUI application does not support interactive functionality, unlike all modern terminal-emulators.
On macOS using zsh as the default shell, for instance, the .zshrc configuration file is not sourced when using the GUI application,
but it will be sourced when using a console instance. In both cases .zshenv is sourced.

The user is advised to always check their $PATH and other critical environment variables.
Any commands not found in the user's $PATH will require absolute paths to function.
For instance, if the user wanted to use gsed, the GNU version of the Unix stream editor sed, even though the user may have installed the gsed package to /usr/local/bin,
for example, and the gsed command functions as expected in the user's terminal or shell scripts, rather than using the command by name:

. global pangram ///

. "All questions asked by five watch experts amazed the judge."

. inshell echo "$pangram" | gsed -E 's/( .....[^a-z] ?)/\U\1/g'

+
+ +-----+

| zsh:1: command not found: gsed | | 127 |

+ stderr
+ +- rc +

use the absolute path of the command

. inshell echo "$pangram" | /usr/local/bin/gsed -E 's/( .....[^a-z] ?)/\U\1/g'

All questions ASKED by five WATCH experts amazed the JUDGE.

If the desired executable is found along the paths listed using:

. inshell echo \$PATH

or command is successfully located by running

. inshell which command

then the function call will proceed. In this particular case echo is a shell built-in and therefore has no direct path to the executable.
inshell allows the user to bypass this limitation with a global macro option that is discussed later in this manual (see Options).

Each time inshell (or shell) is invoked a new shell session is created, the line of commands sent to inshell is executed, and the session is then terminated.
Therefore, any changes made to the environment do not persist.
This limitation means that if the user would like to set a variable's value and then issue subsequent commands based on that variable, the user cannot do so with the following:

. inshell export var=123

. inshell echo \$var

as the value of var will have been lost before the second line is executed.
However, if the user wants to set or export variables to use immediately, they must issue all relevant commands on the same line, with each line of commands separated by semi-colons (;):

. inshell export var=123 ; echo \$var (click to run)

123

for example, or

. global myvar 123

. inshell export var=$myvar ; echo \$var

123

In this fashion Stata macros can be exported to external scripts.
inshell contains two ancillary files to demonstrate this usage which are both POSIX-compatible shell scripts
and which must be installed separately from inshell itself into the PERSONAL system directory.
To install these files (click here) and then (click here).

Using the included shell script named inshell_example1.sh, the contents of which are:

example shell script content

#!/bin/sh
echo $var
echo $((var*2))
var2=$((var/4))
echo $var2
var3=$(date -r 119731017 +%Y-%m-%d)
echo $var3

the user could code:

. global myvar=65536

. inshell export var=$myvar ; . "`: sysdir PERSONAL'"/inshell_example1.sh

( sysdir PERSONAL'"inshell_example1.sh"' :click to run)

65536

131072

16384

1973-10-17

Also included is another shell script named inshell_example2.sh (click to view)
which is intended as a more sophisticated demonstration of shell scripting using inshell. It can be run using the following code:

. inshell export var1=123; export var2=\$var1 ; export var3=$((var1*var2)); export flavor=`= lower(c(edition_real))' ; export os=`c(os)' ; . "`: sysdir PERSONAL'/inshell_example2.sh"

( sysdir PERSONAL'/inshell_example2.sh""':click to run)

Note that the termination of the shell session at the end of every inshell command implies that there is a syntax in use at all times, namely:

inshell cmd1 [; cmd2] [; cmd3] ; exit

The exit in the command above is neither of the Stata commands [R] exit nor [P] exit, but rather the shell's native exit command.

Microsoft PowerShell

inshell will work with Microsoft PowerShell but has not been thoroughly tested on Microsoft Windows. The following are some considerations regarding its use.

When using PowerShell on macOS in inline mode, which is to say issuing commands to PowerShell when it is not the default shell,
single quotes (') are required to surround all dollar sign alphanumeric notation expressions, which as in all cases must also be escaped with a backslash (\).
For instance, to print the $PATH:

. inshell /usr/local/bin/pwsh -Command '\$ENV:PATH'

/usr/local/microsoft/powershell/7:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Stata/utilities/java/macosx-x64/zulu-jdk17.0.5/bin

However, if PowerShell is your default shell (set either by global shell macro S_SHELL or as your system's default shell) they are not required (nor is -Command nor the path to the PowerShell executable).
See S_SHELL global macro for more details.

. inshell \$ENV:PATH

/usr/local/microsoft/powershell/7:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Stata/utilities/java/macosx-x64/zulu-jdk17.0.5/bin

Some other special characters that do not form part of a single expression will often need to be surrounded by single quotes. For instance, to print the $PATH and split it by parsing on a colon (:):

. inshell /usr/local/bin/pwsh -Command '\$ENV:PATH' -split \':\'

or

. inshell /usr/local/bin/pwsh -Command '[System.Environment]::GetEnvironmentVariable('\'PATH\'')' -split \':\'

/usr/local/microsoft/powershell/7

/usr/bin

/bin

/usr/sbin

/sbin

/Applications/Stata/utilities/java/macosx-x64/zulu-jdk17.0.5/bin

In the example above, the colon must be surrounded by single quotes, each of which must be escaped with a backslash.

The following are some examples on how to find the version of PowerShell that is installed.

. inshell /usr/local/bin/pwsh -Command '(Get-Host).Version'

. inshell /usr/local/bin/pwsh -Command '\$host.Version'

. inshell /usr/local/bin/pwsh -Command '\$PSVersionTable.PSVersion'

After each of these commands the version number can be obtained like so:

. display ///

"you are using version `:word 1 of `r(no3)''.`:word 2 of `r(no3)''.`:word 3 of `r(no3)'' of Microsoft PowerShell"

you are using version 7.3.1 of Microsoft PowerShell

Standalone pipes and Math class expressions also require single-quoting, like so:

. inshell /usr/local/bin/pwsh -Command Get-Item env:TMPDIR '|' Select-Object -ExpandProperty Value

. global result 429981696

. inshell /usr/local/bin/pwsh -Command '[math]::Sqrt($result)'

20736

There are likely numerous other potential command structures which also require quoting or escaping, so the user is advised to carefully test each and every inshell command.

Note that when using PowerShell the return code will only reflect the True or False condition of the success of the last operation, which is represented as a boolean of 0 or 1, respectively.

Technical Notes

The output from a shell command can contain control directives called ANSI escape sequences
that are used to specify formatting components such as the color or font styling of text as well as special types of whitespace.

Because the Stata interface is not completely compatible with these sequences, the resulting text that is captured may be garbled or unprintable.
For instance, if shell were used to execute one of the PowerShell commands found above that are used to obtain versioning information:

. shell /usr/local/bin/pwsh -Command '\$PSVersionTable.PSVersion'

the result may look something like this:

[32;1mMajor Minor Patch PreReleaseLabel BuildLabel[0m

[32;1m----- ----- ----- --------------- ----------[0m

7 3 1

The cryptographic sequences at the beginning and end of the first two lines are in this case meant to display the output in bold and green font and should not be visible as text.

Stata does not process these directives as they are intended to be and instead will simply display the coding of the control sequences themselves, however even some of these cannot be printed visibly.
If these control sequences are not removed they will distort the ways in which the information is captured, copied, saved or displayed.

The actual output from the command above does contain invisible characters,
and if the user tries to select the text displayed in the Results window after running the above shell command they may witness the text visibly shift.
Because these control sequences can cause such pervasive aberrations -- in fact too many of them can garble output from subsequent unrelated commands, and can even crash Stata itself --
inshell removes this formatting from all standard output and standard error results. The same command sent through inshell would look like this:

. inshell /usr/local/bin/pwsh -Command '\$PSVersionTable.PSVersion'

Major Minor Patch PreReleaseLabel BuildLabel

----- ----- ----- --------------- ----------

7 3 1

Users should also be aware that lines of inshell commands should not contain any Stata comments as they will be included as part of the command sent to inshell.

It is the author's goal to make inshell compatible with a wide variety of shells and operating systems.
At present (on macOS and Linux) it has been tested with bash, dash, ksh, ksh93, mksh, oil/osh, oksh, tcsh, yash, and zsh,
as well as Microsoft PowerShell, and should also work with ash, csh, and sh.
It is not fully compatible with fish, the "friendly interactive shell."
Although it has not been extensively tested on Microsoft Windows it will work with both Windows Command shell (cmd.exe) and Microsoft PowerShell.

Suggestions

Because of the various limitations noted above, the user is advised to store their commands in an external script and use inshell to execute that script as in syntax 3.
However, such an external script cannot reference Stata macros, unless they are passed to them via an export command (see Environment). For example, if an external script contained the following lines:

example shell script content

#!/bin/sh
echo $PATH
echo \$PATH

and was executed as in syntax 3 the result would look something like this:

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Stata/utilities/java/macosx-x64/zulu-jdk17.0.5/bin

$PATH

as if it were executed directly from the shell in a terminal emulator.

Note that the script file need not be executable in order to function.

When Stata is launched from a terminal emulator, its interface with the shell is an interactive, as opposed to the non-interactive interface in the GUI application,
and therefore it will source the appropriate configuration files, such as .zshrc for zsh and .bashrc for bash.
Despite this, variables defined within .zshenv (for zsh) will be accessible to both the GUI and console instances.

As noted above Stata's implementation of the shell has no interactive abilities and the information printed to the Results window may contain garbled or otherwise unprintable text.
This is because the shell is accessed through Stata's limited shell parser, and therefore the user should explicitly avoid any commands that involve any interactive or complicated display functionality.
inshell attempts to remove any hidden formatting directives that exist as ANSI escape sequences so that the output is clean and legible (see Technical Notes).

Changing directory

inshell also contains a wrapper for cd, allowing the user to change directories while still using inshell.
It is necessary to use a wrapper program to do so because cd operating system commands sent through Stata's shell command will not function.
It confirms the existence of the specified directory beforehand and it even also allows the use of the out-of-date command chdir.
Most importantly, it allows changing directory to paths referenced by variables set within the user's environment.
As in all cases, these variables must be referenced using escaped dollar signs (\$) (see Notation).
For example, if the user wanted to cd to the temporary directory, often set in $TMPDIR:

. inshell cd \$TMPDIR

inshell also respects the differences between the intended behavior of a cd command without arguments on macOS/Linux and Microsoft Windows.
Therefore using inshell cd or even inshell chdir on Windows will simply display the present working directory, and is equivalent to inshell pwd.

Users should note that when using an alternative shell set by global macro S_SHELL,
changing directories to paths specified by shell variables may not function because the alternative shell's configuration files may not be sourced.

Return code

The shell's return code is a numeric code that is used to specify the type of error produced by the commands sent to the shell. It is shell-specific and numerically unrelated to return codes generated by Stata itself.
It can be retrieved after every inshell command which results in an error, and is always an integer > 0.
This value can subsequently be used in error handling routines within Stata do-files.
It is always returned in r(rc), which is not to be confused with Stata's return code, which is always provided in c(rc) or the system variable _rc.

S_SHELL global macro

As of version 2.0 inshell allows the use of the global shell macro S_SHELL.
This macro allows users of shell in all its forms to specify an alternative shell when using shell and thus inshell in Stata.
For more information about S_SHELL see page 793 in the complete PDF manual.

Options

inshell allows its users to set several options that control its behavior through a series of global macros.
It is recommended to set these options within the user's profile.do (see profile) so that they persist across each launch of Stata.

The following two options are currently only available for users of macOS and Linux systems. They can be safely set on Windows systems but will not function.

INSHELL_PATHEXT -- this option allows the user to add a single directory to the beginning of their $PATH.
inshell will check that the directory is valid and export the new extended $PATH before each command that is issued.
Once this is set, absolute paths to commands found along the user's extended $PATH are no longer necessary.

INSHELL_TERM -- this option allows the user to specify an alternative $TERM, as GUI instances of Stata often leave this variable unset.
Valid values for $TERM are often found in /usr/share/terminfo. Some suggested values are "xterm" or "dumb".

If inshell detects that the user is using Microsoft PowerShell the two options above will automatically be disabled, though their values will remain set for future use.

INSHELL_BLOCKEDCMDS -- this option allows the user to add commands to a block list to prevent them from running, as some commands designed for interactive shells will
require Stata to be force quit when they are executed, for example vim or top, which are both included as part of the default list.

Toggle options --
The following options can be toggled on by setting the value of the corresponding global macro to any value.
If the option's name contains the word ENABLE or SET it will enable the option, whereas if it contains DISABLE, setting it will disable the option. All options are toggled off by default.

To unset the option clear the macro.
Because inshell only checks whether or not the macro exists, setting the macro to "ON", "Off", "True", "False", 1, 0, or any other value will have the identical effect of toggling the option on.

INSHELL_ENABLE_AUTOCD -- this option mimics zsh's AUTO_CD feature: if a command is issued that is the name of a sub-directory within the current working directory, it will perform the cd command to that sub-directory.
It is toggled off by default. If this option is set, its functionality will take precedence over otherwise legitimately issued commands.
For instance, if the option is toggled on and the current directory contains a sub-directory named "ls,"
if the user issues the command inshell ls, rather than listing the files in the current directory using the ls command, inshell will issue the cd command to the directory named "ls".
Note that although this function will work in do-files, it is intended for interactive use.

INSHELL_DISABLE_REFOCUS -- After each inshell command focus is redirected to the Results window.
This either may not perform as intended on all systems or may become a nuisance if it is unwanted and therefore this option is offered to disable it.

INSHELL_DISABLE_LONGCMDSMG -- If a command is greater than 300 characters in length, inshell displays a suggestion to store that command as a script file.
By setting this option this message will no longer be displayed.

INSHELL_TAB_SPACES -- This option allows the user to specify the number of spaces into which horizontal tabs (ASCII character code 9) are translated in the standard output.
If it is not set horizontal tabs will remain as such and no conversion to spaces (ASCII character code 32) will occur.
This must be taken into consideration for any string parsing of standard output, as tabs occupy only 1 character length.
Tabs in the standard error are always converted into 4 spaces.

INSHELL_SETSHELL_CSH -- This option must be set if the user's default shell is either csh or tcsh. However, if global macro S_SHELL is set to a csh-type shell, this option is not required.

Diagnostics

inshell contains a program which performs a series of diagnostic routines and which can be with inshell diagnostics.
The screen will be cleared and the current settings of the macro options will be displayed.
If it is set, it will also display the value of the global shell macro S_SHELL.
It will then display which shell is in use along with its version number and the absolute path to the shell. If the system's default shell is a csh-type shell, it will check that the INSHELL_SETSHELL_CSH option is properly set.

It will then display the $PATH, and if the inshell path extension macro option INSHELL_PATHEXT is set,
the existence of the path it contains will be verified and the extended $PATH when using inshell will be displayed.
If the difference between the two $PATHs is equal to the value of INSHELL_PATHEXT, a green check mark will appear.

Though it is not included amongst the displayed option settings, the splash screen displayed at the top of the diagnostics output can be disabled by setting the INSHELL_DISABLE_LOGO macro option.

This diagnostics routine is currently only available for users of macOS or Linux.

Saved results

inshell stores the following in r():

Macros

r(no) the total number of lines of standard output captured. This is 0 when there is no standard output from the command

r(no#) the captured standard output of line #

r(err#) the captured standard error of line #

r(errln) the total number of lines of standard error captured

r(stderr) all of the lines of standard error condensed into a single line

r(rc) the shell's native return code produced by commands, in the event of an error

r(cmd) the command sent to the shell, after all macro expansion occurs. This does not contain any commands related to the global macro options for exporting a new $PATH or $TERM

Often, lines of standard output or standard error can be blank and, because Stata cannot store empty macros, these blank lines cannot be stored in r().
For that reason inshell preserves the sequential numerical ordering of the lines captured into r(no1) to r(nom) or r(err1) to r(errm). If r(no) = m
there will exist m non-empty lines r(no1), ... , r(no(m-1)), r(nom).
However, for completeness, the entire standard output and standard error including blank lines will be stored in the Mata string column vectors M and N, respectively.

Author

Matthew Bryant Hall (mhall@wesleyan.edu)

Also see

Stata:
shell

SSC:
ashell (if installed), windowsmonitor (if installed)