Quick answer
Smoke testing is a quick, shallow set of checks run on each new build or release to confirm that its most critical functions still work before deeper testing or shipping. It is fast, automatable and go or no-go: if any critical path fails, the build goes back.
Formula: Fail if observed < expected − 3 × √expected, where expected = last week's count × this week's sessions ÷ last week's sessions
What is smoke testing?
Smoke testing is a short, shallow set of checks run on a new build or release to answer one question: do the most critical paths still work? If they do, the build is worth testing further or shipping. If any of them fail, you stop right there and send the build back, because deeper testing of a broken build wastes everyone's time.
The name comes from hardware: power on a new circuit board and see whether it starts to smoke. Some writers also trace it to plumbers who pushed smoke through new pipes to look for leaks. Other names for the same idea are build verification test (BVT), confidence test and intake test. Microsoft's engineering guidance describes smoke tests as a quick gate: if the application passes, it is stable enough for further testing or deployment; if it fails, the remaining steps for that version can be stopped.
The core of this entry is one rule: a smoke test is fast, shallow and automatable, and its result is go or no-go. If a check takes an hour, needs judgement or covers edge cases, it belongs in regression testing, not in the smoke test.
Smoke, sanity and regression testing compared
These terms are often mixed up, and some sources treat smoke and sanity as the same thing. Here is a working split:
| Test | When it runs | How deep | Question it answers | Typical size |
|---|---|---|---|---|
| Smoke test | On every new build or release | Shallow, whole product | Does the build work at all? | 5 to 15 checks, minutes |
| Sanity test | After a small change or fix | Narrow, one area | Did this change do what it should without obvious damage? | A handful of checks |
| Regression test | Before a release, often nightly | Deep, wide | Did anything that used to work stop working? | Hundreds to thousands of checks, hours |
Four other glossary terms are about who tests, not how deep:
- Alpha testing is done inside the team on an unfinished product.
- Beta testing puts a nearly finished product in front of real outside users.
- User acceptance testing is the customer or business owner confirming the product meets agreed requirements.
- Canary testing sends a small share of live traffic to the new version in production.
A smoke test can sit in front of any of them: run it on the build before alpha, before a beta drop, before UAT starts, and again on production straight after a canary goes out.
What a good smoke test looks like
- Fast. Minutes, not hours. Pick a budget (for example 10 minutes) and treat it as a hard limit.
- Shallow. One happy path per critical function. No edge cases, no rare browsers.
- Automated where possible. It runs on every build with nobody deciding whether to bother. Manual checks are fine for the first version but tend to be skipped under deadline pressure.
- A hard gate. One failure means no-go. A smoke test where "most of it passed" is ignored teaches the team to ignore it.
- Owned. One named person or team keeps the list current.
Steve McConnell's often-cited description of the daily build says the smoke test should exercise the entire system end to end without being exhaustive, and that it must evolve as the system grows, which is also how it goes wrong (see the mistakes section).
An 8-item smoke checklist
Adapt this to your product; the point is the shape: a few critical user paths, a few critical dependencies, and one check on the data.
- The site or app starts and the home screen loads without errors.
- Sign-up or sign-in works with a test account.
- The core action works: search returns results, a project can be created, a message can be sent.
- Checkout or payment completes in test mode.
- Key API endpoints return success within a set time.
- A write and a read against the database round-trip correctly.
- Critical third parties respond: payment provider callback, email or SMS delivery, login provider.
- Key analytics events arrive: the events that feed revenue and growth numbers, such as
sign_upandpurchase, are still being sent with their required properties.
Eight is a budget, not a target. To add a ninth check, remove one. The last item is the one most teams forget. A release can pass every functional check and still lose the tracking, because a renamed button or a changed page template silently stops an event firing. Nobody sees an error, the dashboards just go quiet. This is why a tracking plan that lists the events that must fire is the natural source for item 8.
Worked example: purchase events 30 minutes after a release
Sample data: a store ships a release on Tuesday at 14:00. The smoke check for item 8 compares purchase events in the first 30 minutes (14:00 to 14:30) with the same weekday and window the week before.
| Last Tuesday 14:00 to 14:30 | This Tuesday 14:00 to 14:30 | |
|---|---|---|
| Sessions | 1,850 | 1,920 |
purchase events | 62 | 31 |
| Purchases per 1,000 sessions | 62 ÷ 1,850 × 1,000 = 33.5 | 31 ÷ 1,920 × 1,000 = 16.1 |
The rate fell by 16.1 ÷ 33.5 − 1 = −51.8%. But is that more than ordinary variation in a 30-minute window? Event counts like this vary roughly like a Poisson count, whose standard deviation is the square root of its mean. Scale last week's count to this week's traffic: expected = 62 × 1,920 ÷ 1,850 = 64.3. The rule below flags a fail when the observed count is more than three standard deviations under expected:
- Threshold = 64.3 − 3 × √64.3 = 64.3 − 24.1 = 40.2.
- Observed 31 < 40.2, so the check fails. (It is (31 − 64.3) ÷ √64.3 ≈ −4.2 standard deviations below expected.)
A fail does not tell you why. Either tracking broke, or purchases really dropped. The next check separates the two: compare against orders in your order system for the same window. If orders are normal and events are half, the release broke tracking; if both are half, the release broke checkout. Both are release-stopping, and the answer decides who gets paged.
The rule needs enough volume. The same method gives useless thresholds when the expected count is small:
| Expected events in the window | Fail below (expected − 3 × √expected) | Gap as share of expected |
|---|---|---|
| 100 | 70.0 | 30% |
| 40 | 21.0 | 47% |
| 20 | 6.6 | 67% |
| 10 | 0.5 | 95% |
With an expected count of 10, only zero events would trigger the rule. For low-volume events, widen the window (a few hours, or compare against the previous day) until expected is at least 30, or use the simpler rule "no event at all in N minutes".
Common mistakes with smoke testing
- Using it as regression. If the smoke test grows to hundreds of cases it stops being a gate. McConnell notes a smoke test can grow to 30 minutes or more as a project grows; set a time budget and move anything beyond it into regression.
- Letting the checklist only grow. Add a check after every incident and you end up with 60 items nobody trusts. Every addition needs a removal.
- Ignoring the data layer. A release that passes all functional checks but breaks
purchasetracking corrupts weeks of reports before anyone notices. - Ignoring partial passes. "Seven of eight passed" is a fail. The point of a gate is that it is binary.
- Running it only in staging. Environment drift means production can fail when staging passes. Run a read-only subset on production after every deploy.
- Treating a pass as a quality verdict. A smoke test says the build is not obviously broken. It says nothing about the 400 less common paths.
Smoke-testing your tracking in TapCub
Take item 8 from the checklist and the purchase example. TapCub does not run your test suite or block a release; it tells you whether the events your release is supposed to send are arriving.
- List the must-fire events in the tracking plan. In All-platform analytics › Data › Tracking plan, record each event with its platforms, trigger, required properties and owner, and set its status to Live once it ships. That list is your item 8.
- After the release, turn on Live verification (24 h). It adds two columns: hits in the last 24 hours and the completeness of required properties. A
purchaseevent at zero hits, or with required properties missing, is your fail. Low completeness is a warning; it does not block incoming data by itself. The 24-hour count is too coarse for a 30-minute comparison like the example, so use it as the quick glance and compare against the order system for the precise check. - Mark the release on the charts. Add an annotation in Configuration › Annotations so any dip in the following hours is read against the release time.
- Put an hourly monitor on the event that matters. Save a custom metric for the event count, then create a monitor in All-platform analytics › Operations › Monitors with a period-change or fixed-threshold condition. A dry run reports how many times it would have fired over the last 30 days. Monitors check hourly at the finest grain, and they only notify; they do not roll anything back.
- Check why data went missing. All-platform analytics › Data › Data quality lists ingestion warnings and collector rejects by reason, such as a domain mismatch or an exclusion rule, so you can tell a tracking bug from a configuration change.
Sources
- Wikipedia, Smoke testing (software): definition, origin of the term, alternative names.
- Microsoft Engineering Fundamentals Playbook, Smoke testing: smoke tests as a fast gate before further testing or deployment.
- Steve McConnell, Daily Build and Smoke Test: the end-to-end smoke test and how it grows.
- NIST/SEMATECH e-Handbook of Statistical Methods, Poisson distribution: mean equals variance, used for the threshold rule.
Was this explanation clear?
We use your answers to fix and expand this page.
Couldn’t send. Please try again in a minute.
Thanks for the feedback. We’ll use it to improve this page.You’ve already sent feedback on this page. Thanks.

Data Engineer
Writes about event design, data quality, and collection workflows, with verifiable implementation methods for reliable analytics.