Free tool Bayesian A/B test calculator
A p-value tells you how surprising the data would be if A and B were the same. This calculator answers the question most people actually ask: how likely is it that B is really better than A, and what do I lose if I pick wrong? Enter visitors and conversions for both versions.
- Beta-binomial model
- Probability B beats A
- Runs locally
How likely is B better than A?
Results are computed by numerical integration of the posterior distributions — deterministic, the same every time, accurate to about 1e-9.
Visitors (or users) in the control.
Visitors in A who converted.
Visitors in the variant.
Visitors in B who converted.
What you assume about the rates before seeing data. Uniform is the usual neutral choice.
Width of the intervals in the table.
Prior “conversions” + 1. Beta(α, β) has mean α ÷ (α + β).
Prior “non-conversions” + 1.
Probability that B beats A
98.2%
Posterior probability that B’s true rate is higher than A’s.
Expected loss if you ship B (points)
0.002
Average conversion you give up if B is in fact worse.
Expected loss if you keep A (points)
0.602
Average conversion you give up if B is in fact better.
Two-sided p-value (z-test)
0.036
The frequentist answer for the same data, for comparison.
Estimates and credible intervals
Equal-tailed intervals at the level you chose. Lift = B ÷ A − 1.
| Metric | Estimate | Low | High |
|---|---|---|---|
| A conversion rate | 4.01% | 3.63% | 4.40% |
| B conversion rate | 4.61% | 4.21% | 5.03% |
| Difference B − A | +0.60 pp | +0.04 pp | +1.16 pp |
| Relative lift | +15.0% | +0.9% | +31.1% |
Share these inputs: the values go after the # in the link, so they never reach a server.
https://tapcub.com/tools/bayesian-ab-test-calculator
Version v1.0Updated Update history
What the outputs mean
Bayesian results are statements about the rates themselves, given the data and the prior — which is what makes them easy to act on.
Probability that B beats A
With 400 of 10,000 converting in A and 460 of 10,000 in B, there is a 98.2% chance that B’s true rate is higher. It does not say how much higher — the interval does.
Expected loss
The average conversion you give up by choosing wrong, weighted by how likely and how bad the mistake is. Shipping B risks 0.002 points; keeping A risks 0.60 points. Many teams ship when the expected loss of the leader falls below a threshold they set in advance.
Credible intervals
Given the data, there is a 95% probability that the lift lies between +0.9% and +31.1%. That range is wide: B is very likely better, but by an uncertain amount.
p-value beside it
The same data gives p = 0.036 with a two-sided z-test. With a flat prior and enough data, P(B beats A) is close to 1 − the one-sided p-value (1 − 0.036 ÷ 2 ≈ 0.982), but the two answer different questions.
The probability is only as good as the data. Check the traffic split with an SRM test first, and decide the sample before you start rather than stopping the moment the probability looks high.
Read a finished test in three steps
Use the counts from the whole test period, for everyone assigned to each group.
-
1
Enter the counts
Visitors and conversions for A (control) and B (variant). Use unique users if you assigned users.
-
2
Choose a prior
Keep the uniform prior unless you have a reason. With thousands of visitors the prior barely matters; with a few dozen it does.
-
3
Decide with probability and loss
Look at P(B beats A) together with the expected loss of shipping B and the interval for the lift, then decide against a threshold set before the test.
Beta-binomial posterior and how this differs from a p-value
Each rate gets a Beta prior. After observing conversions out of visitors, the posterior is again a Beta distribution. All outputs are integrals over the two posteriors, computed numerically — no Monte Carlo sampling, so results do not change between runs.
- The p-value approach asks: if A and B were identical, how surprising is this data? It never gives the probability that B is better
- The Bayesian approach asks: given this data and the prior, how likely is each possible rate? That gives P(B beats A) and expected loss directly
- Both assume independent visitors and a stable rate during the test; neither protects against an uneven split, bots or a broken tracking event
- Peeking still matters: stopping the moment P(B beats A) crosses 95% makes wrong calls more frequent than the number suggests
Posterior
p_A ~ Beta(α₀ + conv_A, β₀ + visitors_A − conv_A)
Uniform prior: Beta(401, 9,601) for A, Beta(461, 9,541) for B
Probability B beats A
P(p_B > p_A) = ∫ f_B(x) · F_A(x) dx
= 98.2% (exact sum: 0.98173)
Expected loss of shipping B
E[max(p_A − p_B, 0)]
= 0.002 percentage points
Same lift, different amounts of data
Uniform prior. Probability and expected loss from this calculator; p-value from a two-sided, pooled two-proportion z-test.
| A (conv / visitors) | B (conv / visitors) | Observed lift | P(B beats A) | Loss if ship B | p-value |
|---|---|---|---|---|---|
| 40 / 1,000 | 46 / 1,000 | +15% | 74.5% | 0.14 pts | 0.51 |
| 400 / 10,000 | 460 / 10,000 | +15% | 98.2% | 0.002 pts | 0.036 |
| 4,000 / 100,000 | 4,100 / 100,000 | +2.5% | 87.2% | 0.006 pts | 0.26 |
The first test is “not significant” but still gives B a 74.5% chance; the expected loss of 0.14 points says acting on it is risky. Calculated with this tool.
How Bayesian results get misused
A Bayesian readout is easier to explain, not immune to bad process.
Reading 95% as “B is 95% better”
P(B beats A) = 95% says B is probably better, not by how much. A 95% chance of a 0.1% lift may not be worth shipping.
Stopping at the first high probability
Checking daily and stopping as soon as the probability crosses a line still inflates wrong decisions. Fix a sample or a decision rule up front.
A strong prior picked after the fact
A custom prior should come from past tests, chosen before you see the result. With small samples, a strong prior can decide the outcome on its own.
Skipping the SRM check
If the split is off, both the probability and the p-value are computed on groups that are no longer comparable.
Getting the counts
TapCub does not split traffic or run experiments — your testing tool does that. If you record the group on each visitor, TapCub can give you visitors and conversions per group.
Break down by variant
Send the group as an event or user property and break reports down by it.
Learn moreConversion per group
A funnel from exposure to conversion, broken down by group, gives the counts for this calculator.
Learn moreBots counted separately
Crawlers never convert; keeping them out of the visitor counts keeps both rates honest.
Learn moreBayesian A/B testing questions
Still have a question?
Chat with the team behind TapCub — we usually reply within a few hours on business days.
What is a Bayesian A/B test?
An A/B test analysed with Bayesian statistics: each conversion rate is described by a probability distribution that is updated with the data. The result is stated directly as the probability that B is better than A, the expected loss of each choice and a credible interval for the lift.
How is “probability to beat control” calculated?
With a uniform prior, A’s rate follows Beta(1 + conversions, 1 + non-conversions), and the same for B. The probability that B beats A is the integral of B’s density times A’s cumulative distribution. This calculator integrates it numerically; for 400/10,000 vs 460/10,000 it is 98.2%.
How is this different from a p-value?
A p-value is the probability of data at least this extreme if there were no difference. It is not the probability that B is better, though it is often read that way. The Bayesian probability is exactly that statement, given the prior. In the default example, p = 0.036 and P(B beats A) = 98.2%.
What is expected loss?
The average amount of conversion you would give up by choosing a version, if it turns out to be the worse one. It combines how likely a mistake is with how costly it would be. A common rule is to ship when the leader’s expected loss is below a small threshold chosen in advance.
Which prior should I use?
Uniform Beta(1, 1) is the usual neutral choice; Jeffreys Beta(½, ½) is another. A custom prior can encode past tests — Beta(4, 96) says “around 4%, worth about 100 visitors of evidence”. With thousands of visitors per group, the choice barely changes the result.
Can I stop a Bayesian test whenever I want?
You can look, but stopping as soon as the probability crosses a threshold makes wrong decisions more frequent than the probability suggests. Plan a sample size or a fixed decision rule, and check the split before deciding.
Is the result random like some Bayesian calculators?
No. Many calculators draw random samples, so the result changes slightly every time. This one integrates the posterior distributions numerically, so the same inputs always give the same output.
Update history
New features, fixes and data refreshes for this tool, newest first.
-
· v1.0
LaunchedFirst release.
Missing something or found an error? Tell us — requests that come up often go into the next update. All tool updates
Plan the next test
Size the next test before it starts, and check its split before reading it.