*! version 17.0 27jun2024

bbandits_initialize Initialize a multi-armed bandit experiment

Syntax

bbandits_initialize [, batch(int 3) arms(int 2) exploration_phase(int 1) sae]

Description

bbandits_initialize sets up the initial data structure for running a multi-armed bandit experiment.
This includes initializing the necessary variables and setting up the experimental conditions.

The program follows the following procedure:

1. It requires a list of all treated units which should be saved in a variable where each row represents a treated unit (see Example 2 below).

2. Then, the program creates the variables reward, chosen_arm, batch and chosen_arm_numeric in Stata and initializes them so that they can be immediately used
in combination with bbandits_update. The variable reward is empty and should be filled during the experiment. chosen_arm contains randomly assigned arms if
an exploration phase was defined, the rest of batches are empty and are filled during the experiment. chosen_arm_numeric generates numeric labels for the chosen_arm
which start from 0 to k (number of arms). It is recommended to use the numeric labels in combination with the bbandits_update command. batch indicates the batches.

3. The program separates all treated units into equal batches based on the original order of the data and the specified number of batches. Hence, if the order is not random it might be recommended
to randomly reorder the data.

4. If an exploration phase is defined, the initialize command will randomly with uniform probability assign the treatment arms for the exploration phase.

5. If the Sequential Arm Elimination (SAE) algorithm is specified, it returns a macro of a numlist of the available arms
which can directly be used for simulating an experiment based on the SAE algorithm (see example below).

Requirements

The underlying calculations are computed in Python. Therefore, Python and the respective Python packages have to be installed. The required Python packages are scikit-learn, numpy,
pandas, scipy, statsmodels, and sfi.
At least Stata 16 is required.

Options

batch(int) specifies the number of batches to divide the experiment into. The default is 3.

arms(int) specifies the number of treatment arms in the bandit experiment. The default is 2.

exploration_phase(int) specifies the number of batches where the algorithm assigns the treatment uniformly. The default value is 1.

sae sets up the optimal data structure for applying the Sequential Arm Elimination algorithm which is presented in Esfandiari et al. (2021, p. 7343).

Examples

Set up the multi-armed bandit experiment with default values:

. bbandits_initialize

Setup with specified number of batches, arms, and exploration phase:

. bbandits_initialize, batches(5) arms(3) exploration_phase(2)

Example 2: Simulated workflow with the Bernoulli Thompson experiment

Clear the dataset and create the required data structure. A dataset in which each row represents a unit that will be treated. A complete list of planned treated units is recommended:

clear

set obs 1000

Create an ID variable:

gen ID = ""

Populate the ID variable with values "school_1", "school_2", ..., "school_1000". Hence there are 1000 schools which are going to be treated:

forval i = 1/1000 {

qui replace ID = "school_" + string(`i') if _n == `i'

}

Initialize the bandit experiment. The "bbandit_initialize" command sets up the optimal data structure for running an experiment with the "bbandits" command.

bbandits_initialize, batches(10) arms(3) exploration_phase(2)

Assign treatment and observe rewards during the exploration phase

generate rand = runiform()

replace reward = .

forval i = 1/2 {

replace reward = 0 if batch == `i'

replace reward = 1 if rand < 0.4 & batch == `i' & chosen_arm_numeric == 2

replace reward = 1 if rand < 0.5 & batch == `i' & chosen_arm_numeric == 1

replace reward = 1 if rand < 0.6 & batch == `i' & chosen_arm_numeric == 0

}

Update assignments using Thompson Sampling:

bbandits_update reward chosen_arm_numeric batch, thompson clipping(0.2)

Run subsequent batches. Capture necessary because in last round of the loop, nothing can be updated anymore.

forval i = 3/10 {

display `i'

replace reward = 0 if batch == `i'

replace reward = 1 if rand < 0.4 & batch == `i' & chosen_arm_numeric == 2

replace reward = 1 if rand < 0.5 & batch == `i' & chosen_arm_numeric == 1

replace reward = 1 if rand < 0.6 & batch == `i' & chosen_arm_numeric == 0

capture bbandits_update reward chosen_arm_numeric batch, thompson clipping(0.2)

}

Inspect final assignments:

bbandits reward chosen_arm_numeric batch

Example 3: Simulated workflow with the Sequential Arm Elimination algorithm

Clear the dataset and create 1,000 observations:

clear

set obs 1000

Create an ID variable:

gen ID = ""

Populate the ID variable with values "school_1", "school_2", ..., "school_1000":

forval i = 1/1000 {

qui replace ID = "school_" + string(`i') if _n == `i'

}

Initialize the bandit experiment using the sequential arm elimination algorithm:

bbandits_initialize, batches(5) arms(3) sae

Display the active arms macro which can be used in the subsequent batches of the bbandits_update command. Alternatively, they can be manually specified:

di "$active_arms_macro"

Generate random rewards:

generate rand = runiform()

replace reward = .

Assign rewards in the first batch:

forval i = 1/1 {

replace reward = 0 if batch == `i'

replace reward = 1 if rand < 0.4 & batch == `i' & chosen_arm == 1

replace reward = 1 if rand < 0.5 & batch == `i' & chosen_arm == 2

replace reward = 1.5 if rand < 0.8 & batch == `i' & chosen_arm == 3

}

Update assignments using sequential arm elimination:

bbandits_update reward chosen_arm_numeric batch, sae active_arms("$active_arms_macro") batch_sae(5)

Assign rewards in the second batch:

forval i = 2/2 {

replace reward = 0 if batch == `i'

replace reward = 1 if rand < 0.4 & batch == `i' & chosen_arm_numeric == 0

replace reward = 1 if rand < 0.5 & batch == `i' & chosen_arm_numeric == 1

replace reward = 1.5 if rand < 0.8 & batch == `i' & chosen_arm_numeric == 2

}

Display updated active arms and update assignments:

bbandits_update reward chosen_arm_numeric batch, sae active_arms("$active_arms_macro") batch_sae(5)

Literature

Esfandiari, H., Karbasi, A., Mehrabian, A., & Mirrokni, V. (2021). "Regret Bounds for Batched Bandits." Proceedings of the AAAI Conference on Artificial Intelligence, 35(8), 7340–7348.

Kemper, J., & Rostam-Afschar, D. (2026). "Earning While Learning: How to Run Batched Bandit Experiments." Stata Journal.

Disclaimer

This software is provided "as is" without warranty of any kind, either expressed or implied. The entire risk as to the quality and performance of the program is with you.

Should the program prove defective, you assume the cost of all necessary servicing, repair, or correction.

In no event will the copyright holders or their employers, or any other party who may modify and/or redistribute this software, be liable to you for damages, including any general, special, incidental,
or consequential damages arising out of the use or inability to use the program.

Authors: Jan Kemper, Davud Rostam-Afschar