help for brain

Neural Network

brain define [if] [in], input(varlist) output(varlist) [hidden(numlist)] [spread(default = 0.25)] [raw | nonorm]

brain norm varlist [if] [in], [raw]

brain reset, [spread(default = 0.25)]

brain save filename

brain load filename

brain signal, [raw]

brain feed input_signals, [raw]

brain train [weight] [if] [in], iter(default = 0) [eta(default = 0.25)] [batch(default = 1)] [report(default = 10)] [best] [noshuffle] [sp]

brain error [weight] [if] [in], [sp]

brain think output_varlist [if] [in], [sp]

brain margin [input_varlist] [weight] [if] [in], [sp]

brain fit original_binary_var [predicted_var] [if] [in], [sp] [th(default = 0.5)]

Description

brain implements a backpropagation network based on the following matrices:

input - defining input variable names, containing the input signal and normalization parameters

output - defining output variable names, containing the output signal and normalization parameters

neuron - containing neuronal signals

layer - defining the structure of the network

brain - containing synapse weights and neuronal bias

Be aware of potential name conflicts! Do not manually change those matrices!

Use matrix list matrix_name to study the content of the network matrices.

Commands and Options

brain

displays the used plugin, version date, the number of processors used for multiprocessing and the structure of the brain matrices. It also verifies the integrity of the structure.

Stored results:

r(plugin) used plugin

r(version) version date

r(mp) number of processors used for multiprocessing

brain define, input(varlist) output(varlist) [hidden(numlist)] [spread(default = 0.25)] [raw | nonorm]

defines the structure of the neural network. Parameters to normalize the input and output variables between [0,1] are determined based on the
active data. The normalization will be skipped if options raw or nonorm are specified. raw still tests that all input and ouput
variables are in the range of [0,1], while [hi:nonorm] just assumes already normalized data without verification. This is usefull, when only selected variables require
normalization (see command norm). The hidden layers can be omitted, leading to a simple perceptron. If specified, every number defines
a hidden layer of the corresponding size starting at the input layer. The starting values of the synapses are randomly distributed between
[-spread, +spread].

brain norm varlist, [raw]

subsequently normalizes specified output and/or input variables separately after an initial definition with define. This is useful to normalize only specific variables, while other
variables, e.g. interdependent shares summing up to 1, stay raw and vice versa. The varlist * can be specified to normalize the whole network. Of course, normalization can be
performed externally without using the brain normalization, but keep in mind, that it has to be consistent for predictions beyond the training data. The input and output matrices
contain the internal normalization parameters.

brain reset, [spread(default = 0.25)]

the weights of an existing network (brain matrix) are re-initialized according to the spread parameter nullifying all training efforts.

brain save filename

save the neural network matrizes into a file. Default postfix is ".brn".

brain load filename

loads the neural network matrizes from a file. Default postfix is ".brn".

brain signal, [raw]

sequentially activates each input signal and reports the difference to the flatline. The signals of the flatline can be found at the bottom of the listing. Specifying raw reports signals
in normalized form [0,1].

Stored results:

r(signal) matrix of input variable on output signals

brain feed input_signals, [raw]

displays the output according to the specified input signals. The option raw declares all values as normalized input signals [0,1], otherwise
they will be interpreted according to the respective variable context. The network status can be accessed using matrix list on the following matrices: input, neuron, output

Stored results:

r(output) matrix of the output signals (de-normalized and raw)

brain train [weight], iter(default = 0) [eta(default = 0.25)] [batch(default = 1)] [report(default = 10)] [best] [noshuffle] [sp]

initiates the backpropagation training for iter iterations using a training factor eta. With the option batch the batch size can
be specified. If it is larger than 1, stochastic gradient descent (SGD) is applied. All gradients, determined by the backpropagation algorithm,
are accumulated to a batch gradient, which is applied to the weights. Batch training may converge faster for deep networks but the real benefit of
batch training is parallelization (see Performance and multiprocessing below). In general, the data is reshuffled before an iteration. If
noshuffle is specified, shuffling will be omitted. This is useful to hone the network in subsequent trainings, as the gradient decent is more
stable along the trodden path and the risk of local minima is mitigated by the initial training based on shuffling. A good strategy is to start
with shuffling (default) and improve the network by subsequent trainings with lower etas and noshuffle. The parameter report
defines the interval for the reporting of the average absolute error and the delta to the previously reported error. Especially for large
etas, the error can fluctuate significantly. The parameter best guarantees that the best network according to the reported error
is used as the final result instead of the last one. This has the benefit of preserving the best network in subsequent training sessions but
requires enough memory to save a copy of the brain matrix.

The training supports weights to adjust for skewed output distributions, i.e. a binary variable with extreme dominance of ones respectively
zeroes.

The function will use multiprocessing unless single processing is enforced with option sp.

Stored results:

r(N) number used observations

r(iter) iteration used for the final brain matrix (see best)

r(err) final error

brain error [weight], [sp]

reports the average absolute error, which is the sum of all absolute errors divided by the number of obs and the number of output variables. The function supports weights.

The function will use multiprocessing unless single processing is enforced with option sp.

Stored results:

r(N) number used observations

r(err) average absolute error

brain think output_varlist, [sp]

creates or overwrites the specified output variables with the prediction of the network. The number of output variables has to match the number
of output neurons. If an input variable exceeds the limits defined for the network, its signal will get truncated accordingly to stay in the range [0,1].

The function will use multiprocessing unless single processing is enforced with option sp.

brain margin [input_varlist] [weight], [sp]

reports the marginal effect of the selected input variables (default = all) on the output variables by calculating the difference between the estimated output and the
output with a constant zero signal for the respective input variable.

The function will use multiprocessing unless single processing is enforced with option sp.

Stored results:

r(margin) matrix of input variables on marginal output signals

brain fit original_binary_var [predicted_var], [sp] [th(default = 0.5)]

reports precision and recall rates for a binary variable based on the prediction of the neural network or a specified predicted variable. The predictied variable can
be omitted for univariate output. This convenient function can also be used ouside the context of brain predictions, i.e. comparing results of probit regressions. Option
th defines the threshold for binary one.

The function will use multiprocessing unless single processing is enforced with option sp.

Stored results:

r(N) number of observations

r(TP) true positives

r(FP) false positives

r(TN) true negatives

r(TP) false negatives

r(Trecall) recall for true: TP/(TP+FN)

r(Frecall) recall for false: TN/(TN+FP)

r(Tprecision) precision for true: TP/(TP+FP)

r(Fprecision) precision for false: TN/(TN+FN)

r(accuracy) accurary: (TP+TN)/(TP+TN+FP+FN)

r(threshold) threshold for binary one

Normalization

An Artificial Neural Network (ANN) simulates very roughly the signal processing in the brain. Therefore, it is essential that all
signals passing through the brain matrix have a limited strength to prevent overloading. In this implementation, the signals are always
in the range between 0 and 1. During the definition of the brain with the command define, the input and output matrices
contain normalization parameters based on the active training data to provide appropriate normalization not only for training but also
for later predictions. Sometimes, data is already within the valid range, which can be declared with the parameter raw during
definition. If only specific variables require normalization, it can be skipped with the parameter nonorm. The command norm
allows for subsequent normalizations of groups of variables to enable a mix of normalized and raw variables. Skipping normalization is
only required, if variables are interdependent and a column wise normalization may distort those row wise relations, i.e. shares. In
those cases, the column wise normalization has to be provided by the user.

Compatibility and Multiprocessing

Except for the margin and signal commands brain fully supports older STATA Versions (tested for STATA 13). The before mentioned commands still work for older version
but are limited to a maximum of 11000 input respectively output variables.

Given the unclear situation of the inclusion of openmp of the different UNIX distributions including MacOS, the corresponding plugins do not support
multiprocessing. Only the Windows plugin benefits from multiprocessing. Please consult the Github page of the author (see below) to enable multiprocessing for the other OS
by locally compiling the source.

Performance and Multiprocessing

brain uses C plugins supporting multiprocessing (the parallel usage of multiple processors/cores). While some of the commands, e.g. error
or think, always benefit from multiple cores, the train command uses multiprocessing only if a batch size larger than 1 is
specified. During training, the observations of a batch are distributed to separate cores. This is possible because the delta vectors derived
from every observation in a batch are independent. Only after the processing of a batch, the vectors will be applied to the weights of the neural
network. By default, the training will use all available processors unless the batch size is smaller than the processor number. For the highest
efficiency, the batch size should be divisible by the number of processors, which can be inquired by calling brain without an option.
Multiprocessing can be deactivated with the option sp (single processing). This can be useful for keeping the etiquette on shared servers or
if some leeway for other applications is needed. If a neural network has only a low number of weights, the overhead imposed by multiprocessing can
lead to a negative effect on performance, especially on machines with only 4 or less cores. It is advised to undertake performance comparisons
between single- and multiprocessing (see Example 2).

Example 1: OLS vs brain on unobserved interaction and polynomials

clear
set obs 100
gen x1 = invnorm(uniform())
gen x2 = invnorm(uniform())
gen y = x1 + x2 + x1^2 + x2^2 + x1*x2

sum y
scalar ymean = r(mean)
egen sst = sum((y-ymean)^2)

reg y x1 x2

predict yreg
egen rreg = sum((y-yreg)^2)

brain define, input(x1 x2) output(y) hidden(10 10)
brain train, iter(1000) eta(1) sp
brain think ybrain

egen rbrain = sum((y-ybrain)^2)

di "R-squared reg: " 1-rreg/sst
di "R-sq. brain: " 1-rbrain/sst

Example 2: Single- vs. muliprocessing

clear
set more off
set obs 2000
gen x1 = invnorm(uniform())
gen x2 = invnorm(uniform())
gen y = x1 + x2 + x1^2 + x2^2 + x1*x2

timer clear
global hidden1 = "10 10"
global hidden2 = "50 50"

brain define, input(x1 x2) output(y) hidden($hidden1)
brain save test

timer on 1
brain train, iter(500) eta(0.25) batch(16) sp
timer off 1

brain load test
timer on 2
brain train, iter(500) eta(0.25) batch(16)
timer off 2

brain define, input(x1 x2) output(y) hidden($hidden2)
brain save test

timer on 3
brain train, iter(500) eta(0.25) batch(16) sp
timer off 3

brain load test
timer on 4
brain train, iter(500) eta(0.25) batch(16)
timer off 4

timer list

Example 3: OLS vs brain on non-linear function

clear
set obs 200
gen x = 4*_pi/200 *_n
gen y = sin(x)

reg y x
predict yreg

brain define, input(x) output(y) hidden(20)
brain train, iter(1000) eta(1) sp
brain think ybrain

twoway (scatter y x, sort) (line yreg x, sort) (line ybrain x, sort)

Example 4: probit vs brain on binary output variable and application of weights

sysuse auto, clear
tab foreign

probit foreign price-gear_ratio
predict pforeign
brain fit foreign pforeign

brain define, input(price-gear_ratio) output(foreign) hidden(20 20)
brain train, iter(500) eta(1) best sp
brain fit foreign

keep if foreign == 0 | make == "Audi Fox" // extremely skewed distribution (only one foreign car)
tab foreign

brain define, input(price-gear_ratio) output(foreign) hidden(10)
brain train, iter(500) eta(1) best sp // not weighted: predicting the outlier provides almost no benefit
brain fit foreign

gen w = 1
sum foreign
replace w = (1-r(mean))/r(mean) if foreign == 1

brain define, input(price-gear_ratio) output(foreign) hidden(10)
brain train [pweight=w], iter(500) eta(1) best sp // weighted: outlier has a large impact on the error
brain fit foreign

Example 5: categorical respectively ordinal variables

sysuse auto, clear
gen byte t = weight/1000
tab t, gen(t)
brain define, input(price-trunk length-foreign) output(t1-t4) hidden(20 20)
brain train, iter(500) eta(1) best sp
brain think b1 b2 b3 b4
brain fit t1 b1
brain fit t2 b2
brain fit t3 b3
brain fit t4 b4

Example 6: dynamic adjustment of eta during training exploiting the "best" option

clear
set obs 100
gen x1 = invnorm(uniform())
gen x2 = invnorm(uniform())
gen y = x1 + x2 + x1^2 + x2^2 + x1*x2

sum y
scalar ymean = r(mean)
egen sst = sum((y-ymean)^2)

reg y x1 x2
predict yreg
egen rreg = sum((y-yreg)^2)

brain define, input(x1 x2) output(y) hidden(10 10)
local eta = 20 // For demonstration only. If eta is too large, the training can freeze. Usually you start with 1.
local half = 1
local run = 1
while `run' <= 50 & `half' <= 10 { // exit after 50 improving iteration cycles or after 10 eta halvings
di as text "RUN " as result `run'
brain train, iter(100) eta(`eta') best sp
if r(iter) == 0 { // no improvement to the beginning of the cycle
local eta = `eta'/2
local half = `half'+1
}
else {
local run = `run'+1
}
}
brain think ybrain
egen rbrain = sum((y-ybrain)^2)

di "R-squared reg: " 1-rreg/sst
di "R-sq. brain: " 1-rbrain/sst

Update History

2021.19.05 The new norm command allows for specific normalization of groups of variables.

The nonorm option of the define command skips nromalization and testing.

New command reset re-initilializes the weights of the brain without redefining the normalization.

Default spread set from 0.5 to 0.25, as specified in the help file.

2021.16.02 Recompiled the MAC plugin to exclude openmp libraries because of incompatible distributions.

2020.12.08 Improved compatibility with older STATA versions (below 15) in terms of matrix sizes.

Fixed a bug that prevented random weight initialization.

Recompiled the UNIX plugin to exclude openmp libraries because of incompatible distributions.

2020.11.30 Fixed a bug that prevented the usage of the fit command without a second variable.

The fit function allows the specification of a threshold for binary one.

2020.10.07 The define paramter raw prevents automatic normalization for already normalized data.

2020.04.28 Revised documentation and program in regard of proper usage of weights.

2020.03.11 All matrices can exceed matsize limitations by taking a detour over mata.

The brain matrix fragmentation of the former solution (2020.03.02) is obsolete.

The brain, load and save commands additionally verify the integrity of the matrix structure.

2020.03.02 Replaced all mata components with C plugins supporting multiprocessing.

The network can exceed the maximum matrix size of Stata.

Syntax errors are now more consistent.

2019.11.13 The commands train and error support weights.

New command fit calculates recall and precision for binary output.

2019.03.22 Implemented batch training.

Changed training ouput to interval reports of real absolute errors instead of intermediate errors.

nosort is now called noshuffle as shuffling is now applied before every iteration.

The best option alway picks the best intermediate result in case of alternating errors.

New command error reports the overall absolute error instead of using the train command with iter(0).

signal now works as intended.

2018.11.21 Initial version.

Acknowledgements

I thank my colleague Robert Braeutigam for facilitating the Unix and Mac plugin versions.

Author

Thorsten Doherr

Centre for European Economic Research

L7,1

68161 Mannheim

Germany

E-Mail: doherr@zew.de

Internet: https://www.zew.de

Github: https://github.com/ThorstenDoherr/brain