What is sequential testing? Peeking, methods and an example

What is sequential testing?

Sequential testing is a way of analysing an experiment repeatedly while data comes in, with stopping rules built so the overall false positive rate stays at the level you chose, typically 5%. You can look at an A/B test every day and stop early when the evidence is strong, without fooling yourself.

The alternative, a fixed-horizon test, is analysed exactly once, at a sample size calculated in advance from the minimum detectable effect and statistical power. That works, as long as nobody looks early. In practice, almost everybody looks.

The peeking problem, in numbers

If a test has no real effect and you check it once, a 5% significance threshold gives a 5% chance of a false positive. If you check it repeatedly and stop the first time p < 0.05, every look is another chance for random noise to cross the line. We simulated this for equally spaced looks (40,000 runs per row):

Checking daily for three weeks and stopping on the first "significant" day means roughly one in four winning tests is noise. Armitage and colleagues described this effect in 1969; it is a form of p-hacking that happens without any bad intent.

Sequential testing methods

Many experimentation platforms now ship one of these by default. The important thing is to choose the method before the test starts.

A worked example: O'Brien-Fleming with five looks

Sample test: a new pricing page vs the old one, reviewed weekly for up to five weeks. For five equally spaced looks at α = 0.05 (two-sided), the O'Brien-Fleming boundaries are 4.56, 3.23, 2.63, 2.28 and 2.04.

The boundaries are strict early, when the estimate is noisy, and close to the usual 1.96 at the end, so the test loses very little power compared with a single analysis. At look 2 the naive p-value was already below 0.05; stopping there would have been exactly the peeking error. At look 3 the evidence crossed a boundary designed for repeated looks, so stopping is legitimate.

Trade-offs

• Faster decisions on big effects; small effects still need the full sample.
• Slightly larger maximum sample size than a fixed test with the same power.
• Effect estimates are biased upward when a test stops early, because it stopped on a high point. Report the estimate with that caveat, or use the method's adjusted estimate.
• Planning is still required: the looks, method and maximum sample size must be set in advance.

When to use it

Use sequential testing when people will look at results before the end (they will), when stopping early on a harmful change matters, or when traffic is expensive. Use a fixed-horizon test when you can genuinely commit to one analysis. If the goal is to earn during the test rather than to measure an effect, a multi-armed bandit is a different tool.

Sequential testing and TapCub

TapCub does not run experiments or compute experiment statistics. If your testing tool assigns variants, send the variant as an event property (for example tapcub.track('pricing_view', { variant: 'B' })) and break down conversion, revenue and retention by it in event analysis. That shows what happened downstream; the stopping decision should follow the sequential method your experiment tool uses. Background on significance itself is in statistical significance and p-value.

Sources

• Wald, A. (1945). Sequential Tests of Statistical Hypotheses. Annals of Mathematical Statistics, 16(2), 117–186.
• Armitage, P., McPherson, C. K. and Rowe, B. C. (1969). Repeated Significance Tests on Accumulating Data. Journal of the Royal Statistical Society A, 132(2), 235–244.
• O'Brien, P. C. and Fleming, T. R. (1979). A Multiple Testing Procedure for Clinical Trials. Biometrics, 35(3), 549–556.
• Johari, R., Pekelis, L. and Walsh, D. (2017). Always Valid Inference: Bringing Sequential Analysis to A/B Testing.