Free tool SRM checker: sample ratio mismatch test
Before you trust an A/B test result, check the split. If you configured 50/50 but the groups ended up 50.6/49.4 on a hundred thousand users, something removed or added users unevenly — and the result can no longer be trusted. Paste the users per group to find out.
- Chi-square goodness of fit
- Threshold p < 0.0005
- Runs locally
Did each group get its share?
One group per line: an optional name, then the number of users (or sessions — whatever unit you randomised on). Thousands separators are fine; no spaces inside the number.
2 to 6 lines, for example “Control 50,123”.
50/50, 1:1:1, 40/30/30… Leave empty for an equal split.
Flag SRM when the p-value is below this.
Verdict
- SRMSample ratio mismatch: p = 2.72 × 10⁻⁴ is below the threshold of 0.0005. Do not read the test result until you find the cause.
- Check“Variant” has 1.16% fewer users than expected — the group that most likely lost users.
p-value
2.72 × 10⁻⁴
Probability of a split at least this uneven if assignment worked as configured.
Chi-square (χ²)
13.252
Degrees of freedom
1
Groups − 1.
Total users
99,100
By group
Deviation = (observed − expected) ÷ expected.
| Group | Observed | Expected | Observed share | Expected share | Deviation | χ² part |
|---|---|---|---|---|---|---|
| Control | 50,123 | 49,550 | 50.58% | 50.00% | +1.16% | 6.63 |
| Variant | 48,977 | 49,550 | 49.42% | 50.00% | −1.16% | 6.63 |
Likely causes to check
Grouped after Microsoft’s SRM taxonomy (Fabijan et al.): assignment, execution, logging and analysis.
- Possible causeBots: crawlers or scripts reach one variant more than the other, or bot filtering runs after assignment and removes more users from one group.
- Possible causeRedirects: in a redirect test, visitors who leave during the extra hop, or whose tracking fires before it, are lost from the redirected group only.
- Possible causeCaching: a cached or CDN-served page shows the same variant to many visitors, or the assignment cookie is not part of the cache key.
- Possible causeAssignment bugs: faulty hashing or randomisation, users re-bucketed after logging in or switching devices, or IDs that do not match between systems.
- Possible causeRamp-up: the split was changed during the test (10% → 50%, say) and the analysis mixes the periods.
- Possible causeLogging and analysis: the variant changes whether the exposure event fires (slower page, error), a data join drops rows, or a filter applied after assignment removes users unevenly.
Share these inputs: the values go after the # in the link, so they never reach a server.
https://tapcub.com/tools/srm-checker
Version v1.0Updated Update history
What the checker tells you
An SRM check does not say whether the variant is better. It says whether the comparison is fair.
p-value
How likely a split this uneven would be if users had been assigned exactly as configured. 50,123 vs 48,977 looks close, but on 99,100 users it has p = 0.00027 — about 1 in 3,700.
Verdict
SRM when p is below the threshold. The default 0.0005 is the conservative cut-off Microsoft’s Experimentation Platform uses to keep false alarms rare; real SRMs usually produce far smaller p-values.
Deviation by group
Which group is short or over, as a share of its expected size. A group that is short usually lost users through a redirect, a crash or a filter; a group that is over gained bots or duplicates.
χ² part
Each group’s contribution to the chi-square statistic. With three or more groups, the largest part points at the group to investigate first.
A clean SRM check does not prove the experiment is bug-free; it only rules out one common and serious class of problems.
Check the split in three steps
Run the check before looking at any conversion numbers — ideally every day while the test runs.
-
1
Count the randomisation unit
Users per group if you assigned users, sessions if you assigned sessions. Use the counts of everyone assigned, not only those who converted.
-
2
Enter the configured split
The split from your testing tool: 50/50, 34/33/33 or whatever you set. Leave it empty for an equal split.
-
3
Read the verdict
SRM: stop and investigate the listed causes before analysing anything. No SRM: go on to compare the conversion rates.
Chi-square goodness-of-fit test
Expected counts come from the total and the configured split. The chi-square statistic sums the squared differences, and its p-value comes from the chi-square distribution with groups − 1 degrees of freedom (computed with the regularized incomplete gamma function, accurate to about 1e-12).
- Expected count Eᵢ = total users × configured share of group i
- Assumes users are assigned independently; repeat visits by the same user must be counted once
- Needs every expected count of at least 5 for the approximation to hold — the checker warns below that
- Threshold conventions differ: Microsoft ExP uses p < 0.0005; some tools use 0.001 or 0.01. Lower thresholds mean fewer false alarms when you check every day
Expected count
Eᵢ = N × wᵢ
99,100 × 50% = 49,550
Statistic
χ² = Σ (Oᵢ − Eᵢ)² ÷ Eᵢ
573² ÷ 49,550 + 573² ÷ 49,550 = 13.25
p-value
p = P(χ²ₖ₋₁ ≥ χ²) = Q((k − 1)/2, χ²/2)
df = 1: p = 2.72 × 10⁻⁴ < 0.0005 → SRM
Small imbalances on large samples
The same relative imbalance is harmless on a small test and alarming on a large one. Configured split 50/50.
| Group A | Group B | Share of A | p-value | Verdict at 0.0005 |
|---|---|---|---|---|
| 1,000 | 1,100 | 47.6% | 0.029 | No SRM |
| 10,000 | 10,100 | 49.75% | 0.48 | No SRM |
| 100,000 | 101,000 | 49.75% | 0.026 | No SRM |
| 50,123 | 48,977 | 50.58% | 0.00027 | SRM |
A 47.6% share on 2,100 users is not an SRM; a 50.58% share on 99,100 users is. Calculated with this tool.
How SRM gets missed
SRM is one of the most common reasons an A/B test result is wrong, and one of the easiest to check.
Eyeballing the split
50.6/49.4 looks fine. Whether it is fine depends on the sample size, which is why the check needs a statistical test.
Checking after filtering
Run the check on everyone assigned. Counting only users who reached a later step mixes the variant’s effect into the split.
Counting the wrong unit
If you assign users but count page views, heavy users in one group look like an SRM. Count the unit you randomised on.
Re-weighting instead of fixing
Scaling the groups back to 50/50 hides the symptom but not the bias: the users who went missing were probably not random.
Two common SRM causes TapCub helps you see
TapCub does not assign variants or run experiments. It can help you find two causes of SRM: bot traffic and uneven drop-off between groups.
Bots counted separately
Search engines, AI crawlers and scrapers are split from people, so bot traffic in one group shows up instead of hiding in the totals.
Learn moreUsers by group
Record the group as an event or user property and break reports down by it to compare counts per group.
Learn moreWhere users drop
A funnel broken down by group shows whether one variant loses users before the exposure event fires.
Learn moreSample ratio mismatch questions
Still have a question?
Chat with the team behind TapCub — we usually reply within a few hours on business days.
What is a sample ratio mismatch (SRM)?
A difference between the traffic split you configured and the split you observed that is too large to be chance. It means something added or removed users unevenly, so the groups are no longer comparable.
How do I check for SRM?
Run a chi-square goodness-of-fit test on the users per group against the configured split. For 50,123 vs 48,977 on a 50/50 split, χ² = 13.25 with 1 degree of freedom, p = 0.00027.
What p-value threshold should I use for SRM?
Microsoft’s Experimentation Platform uses p < 0.0005, chosen to be conservative so that daily checks rarely raise false alarms. Some tools use 0.001 or 0.01. Real SRMs usually show p-values far below all of them.
What causes SRM?
Microsoft’s taxonomy groups the causes by stage: assignment (bad hashing, users re-bucketed), execution (redirects, a variant that crashes or loads slower), log processing (joins that drop rows, bot filtering), analysis (filters applied after assignment) and changes made by the experimenters, such as an uneven ramp-up.
Can I still use the results if there is an SRM?
Generally no. The users who went missing are rarely random, so any lift you measure may come from who was lost rather than from the change. Find the cause, fix it and re-run.
Should I check users or sessions?
The unit you randomised on. Most web tests assign users (via a cookie or ID), so count unique users per group.
Is this the same as a chi-square test?
It is the goodness-of-fit form of it: observed counts against a known expected split. The chi-square calculator also handles contingency tables, such as conversion by group.
Update history
New features, fixes and data refreshes for this tool, newest first.
-
· v1.0
LaunchedFirst release.
Missing something or found an error? Tell us — requests that come up often go into the next update. All tool updates
Split is fine? Read the result
With a clean split, compare the conversion rates — with a p-value or as a probability that B beats A.