What is a multi-armed bandit? Algorithms, examples and vs A/B testing

What is a multi-armed bandit?

A multi-armed bandit is a way of testing several options while the test is running and earning: instead of splitting traffic evenly until the end, it gradually sends more visitors to the options that are performing better. It balances exploration (trying options to learn which is best) with exploitation (using the best option found so far).

The name comes from slot machines, nicknamed "one-armed bandits". Picture a row of machines with unknown payout rates. Pull only the one that paid first and you may miss a better one; keep pulling all of them equally and you waste money on the bad ones. Bandit algorithms decide how to split the pulls.

Exploration, exploitation and regret

The cost of testing is called regret: the reward you missed by not always choosing the best option.

Example (sample data). Two landing pages convert at 5% and 6%. A classic A/B test sends 10,000 visitors 50/50:

• 5,000 × 5% + 5,000 × 6% = 250 + 300 = 550 conversions
• Sending everyone to the better page would have produced 10,000 × 6% = 600
• Regret = 600 − 550 = 50 conversions

A bandit reduces that regret by shifting traffic toward the 6% page as evidence builds up. The price is that the weaker page gets fewer visitors, so you learn less precisely how much worse it is.

Common bandit algorithms

A worked example: Thompson sampling

Sample data: three email headlines after the first 1,000 sends.

Thompson sampling models each rate as a Beta distribution: Beta(1 + conversions, 1 + non-conversions). Drawing from the three distributions many times (we ran 200,000 draws), C has the highest draw 71.3% of the time, B 23.9% and A 4.9%. So the next batch is split roughly 71 / 24 / 5. B still gets meaningful traffic because it could plausibly be the best; A is nearly retired but not ruled out. After each batch the distributions are updated and the split recalculated.

Multi-armed bandit vs A/B testing

Rule of thumb: use a bandit to earn, an A/B test to learn. If the variant will live for years, the learning is worth more than a few days of extra conversions.

Contextual bandits

A contextual bandit also uses information about the visitor (device, country, new vs returning) to choose. It can learn that headline B works on mobile and C on desktop. It needs much more traffic and careful feature design, and it behaves like a recommendation system more than a test.

Pitfalls

• Delayed conversions. If purchases arrive days after the click, the bandit allocates on incomplete data and over-rewards whatever converts fast.
• Changing behaviour. Weekday vs weekend audiences convert differently. When allocation changes over time, pooled results can mislead, a form of Simpson's paradox.
• No clean inference. Standard p-values assume fixed allocation. If you need significance, use an A/B test or a method built for adaptive designs such as sequential testing.
• Too many arms, too little traffic. With ten options and 500 visitors a week, nothing converges.

Multi-armed bandits and TapCub

TapCub is an analytics tool; it does not allocate traffic, run A/B tests or bandits. If your testing or email tool runs the bandit, send the variant as an event property, for example tapcub.track('headline_view', { variant: 'C' }), and break down the conversion event by that property in event analysis. That gives you an independent check on what the bandit reports, including downstream outcomes it cannot see, such as retention.

Sources

• Thompson, W. R. (1933). On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples. Biometrika, 25(3/4), 285–294.
• Auer, P., Cesa-Bianchi, N. and Fischer, P. (2002). Finite-time Analysis of the Multiarmed Bandit Problem. Machine Learning, 47, 235–256.
• Russo, D. et al. (2018). A Tutorial on Thompson Sampling. Foundations and Trends in Machine Learning.
• Slivkins, A. (2019). Introduction to Multi-Armed Bandits.