TapCub is live — analytics, insights and live chat, free on one platform
Glossary

What is smoke testing? Definition, 8-item checklist and a release-day example

Smoke testing is a quick, shallow check that a new build's most critical paths still work. How it differs from sanity and regression testing, an 8-item checklist, and a worked check of purchase events after a release.

Lucas Hayes Reviewed by Chen Yu Also called smoke test Last updated Nov 17, 2026 Glossary

Quick answer

Smoke testing is a quick, shallow set of checks run on each new build or release to confirm that its most critical functions still work before deeper testing or shipping. It is fast, automatable and go or no-go: if any critical path fails, the build goes back.

Formula: Fail if observed < expected − 3 × √expected, where expected = last week's count × this week's sessions ÷ last week's sessions

What is smoke testing?

Smoke testing is a short, shallow set of checks run on a new build or release to answer one question: do the most critical paths still work? If they do, the build is worth testing further or shipping. If any of them fail, you stop right there and send the build back, because deeper testing of a broken build wastes everyone's time.

The name comes from hardware: power on a new circuit board and see whether it starts to smoke. Some writers also trace it to plumbers who pushed smoke through new pipes to look for leaks. Other names for the same idea are build verification test (BVT), confidence test and intake test. Microsoft's engineering guidance describes smoke tests as a quick gate: if the application passes, it is stable enough for further testing or deployment; if it fails, the remaining steps for that version can be stopped.

The core of this entry is one rule: a smoke test is fast, shallow and automatable, and its result is go or no-go. If a check takes an hour, needs judgement or covers edge cases, it belongs in regression testing, not in the smoke test.

Smoke, sanity and regression testing compared

These terms are often mixed up, and some sources treat smoke and sanity as the same thing. Here is a working split:

TestWhen it runsHow deepQuestion it answersTypical size
Smoke testOn every new build or releaseShallow, whole productDoes the build work at all?5 to 15 checks, minutes
Sanity testAfter a small change or fixNarrow, one areaDid this change do what it should without obvious damage?A handful of checks
Regression testBefore a release, often nightlyDeep, wideDid anything that used to work stop working?Hundreds to thousands of checks, hours

Four other glossary terms are about who tests, not how deep:

  • Alpha testing is done inside the team on an unfinished product.
  • Beta testing puts a nearly finished product in front of real outside users.
  • User acceptance testing is the customer or business owner confirming the product meets agreed requirements.
  • Canary testing sends a small share of live traffic to the new version in production.

A smoke test can sit in front of any of them: run it on the build before alpha, before a beta drop, before UAT starts, and again on production straight after a canary goes out.

What a good smoke test looks like

  • Fast. Minutes, not hours. Pick a budget (for example 10 minutes) and treat it as a hard limit.
  • Shallow. One happy path per critical function. No edge cases, no rare browsers.
  • Automated where possible. It runs on every build with nobody deciding whether to bother. Manual checks are fine for the first version but tend to be skipped under deadline pressure.
  • A hard gate. One failure means no-go. A smoke test where "most of it passed" is ignored teaches the team to ignore it.
  • Owned. One named person or team keeps the list current.

Steve McConnell's often-cited description of the daily build says the smoke test should exercise the entire system end to end without being exhaustive, and that it must evolve as the system grows, which is also how it goes wrong (see the mistakes section).

An 8-item smoke checklist

Adapt this to your product; the point is the shape: a few critical user paths, a few critical dependencies, and one check on the data.

  1. The site or app starts and the home screen loads without errors.
  2. Sign-up or sign-in works with a test account.
  3. The core action works: search returns results, a project can be created, a message can be sent.
  4. Checkout or payment completes in test mode.
  5. Key API endpoints return success within a set time.
  6. A write and a read against the database round-trip correctly.
  7. Critical third parties respond: payment provider callback, email or SMS delivery, login provider.
  8. Key analytics events arrive: the events that feed revenue and growth numbers, such as sign_up and purchase, are still being sent with their required properties.

Eight is a budget, not a target. To add a ninth check, remove one. The last item is the one most teams forget. A release can pass every functional check and still lose the tracking, because a renamed button or a changed page template silently stops an event firing. Nobody sees an error, the dashboards just go quiet. This is why a tracking plan that lists the events that must fire is the natural source for item 8.

Worked example: purchase events 30 minutes after a release

Sample data: a store ships a release on Tuesday at 14:00. The smoke check for item 8 compares purchase events in the first 30 minutes (14:00 to 14:30) with the same weekday and window the week before.

Last Tuesday 14:00 to 14:30This Tuesday 14:00 to 14:30
Sessions1,8501,920
purchase events6231
Purchases per 1,000 sessions62 ÷ 1,850 × 1,000 = 33.531 ÷ 1,920 × 1,000 = 16.1

The rate fell by 16.1 ÷ 33.5 − 1 = −51.8%. But is that more than ordinary variation in a 30-minute window? Event counts like this vary roughly like a Poisson count, whose standard deviation is the square root of its mean. Scale last week's count to this week's traffic: expected = 62 × 1,920 ÷ 1,850 = 64.3. The rule below flags a fail when the observed count is more than three standard deviations under expected:

  • Threshold = 64.3 − 3 × √64.3 = 64.3 − 24.1 = 40.2.
  • Observed 31 < 40.2, so the check fails. (It is (31 − 64.3) ÷ √64.3 ≈ −4.2 standard deviations below expected.)

A fail does not tell you why. Either tracking broke, or purchases really dropped. The next check separates the two: compare against orders in your order system for the same window. If orders are normal and events are half, the release broke tracking; if both are half, the release broke checkout. Both are release-stopping, and the answer decides who gets paged.

The rule needs enough volume. The same method gives useless thresholds when the expected count is small:

Expected events in the windowFail below (expected − 3 × √expected)Gap as share of expected
10070.030%
4021.047%
206.667%
100.595%

With an expected count of 10, only zero events would trigger the rule. For low-volume events, widen the window (a few hours, or compare against the previous day) until expected is at least 30, or use the simpler rule "no event at all in N minutes".

Common mistakes with smoke testing

  • Using it as regression. If the smoke test grows to hundreds of cases it stops being a gate. McConnell notes a smoke test can grow to 30 minutes or more as a project grows; set a time budget and move anything beyond it into regression.
  • Letting the checklist only grow. Add a check after every incident and you end up with 60 items nobody trusts. Every addition needs a removal.
  • Ignoring the data layer. A release that passes all functional checks but breaks purchase tracking corrupts weeks of reports before anyone notices.
  • Ignoring partial passes. "Seven of eight passed" is a fail. The point of a gate is that it is binary.
  • Running it only in staging. Environment drift means production can fail when staging passes. Run a read-only subset on production after every deploy.
  • Treating a pass as a quality verdict. A smoke test says the build is not obviously broken. It says nothing about the 400 less common paths.

Smoke-testing your tracking in TapCub

Take item 8 from the checklist and the purchase example. TapCub does not run your test suite or block a release; it tells you whether the events your release is supposed to send are arriving.

  1. List the must-fire events in the tracking plan. In All-platform analytics › Data › Tracking plan, record each event with its platforms, trigger, required properties and owner, and set its status to Live once it ships. That list is your item 8.
  2. After the release, turn on Live verification (24 h). It adds two columns: hits in the last 24 hours and the completeness of required properties. A purchase event at zero hits, or with required properties missing, is your fail. Low completeness is a warning; it does not block incoming data by itself. The 24-hour count is too coarse for a 30-minute comparison like the example, so use it as the quick glance and compare against the order system for the precise check.
  3. Mark the release on the charts. Add an annotation in Configuration › Annotations so any dip in the following hours is read against the release time.
  4. Put an hourly monitor on the event that matters. Save a custom metric for the event count, then create a monitor in All-platform analytics › Operations › Monitors with a period-change or fixed-threshold condition. A dry run reports how many times it would have fired over the last 30 days. Monitors check hourly at the finest grain, and they only notify; they do not roll anything back.
  5. Check why data went missing. All-platform analytics › Data › Data quality lists ingestion warnings and collector rejects by reason, such as a domain mismatch or an exclusion rule, so you can tell a tracking bug from a configuration change.

Sources

  • Wikipedia, Smoke testing (software): definition, origin of the term, alternative names.
  • Microsoft Engineering Fundamentals Playbook, Smoke testing: smoke tests as a fast gate before further testing or deployment.
  • Steve McConnell, Daily Build and Smoke Test: the end-to-end smoke test and how it grows.
  • NIST/SEMATECH e-Handbook of Statistical Methods, Poisson distribution: mean equals variance, used for the threshold rule.

Was this explanation clear?

We use your answers to fix and expand this page.

Lucas Hayes

Data Engineer

Writes about event design, data quality, and collection workflows, with verifiable implementation methods for reliable analytics.

FAQ

Questions people ask

Short answers to the follow-up questions that come up most.

Still have a question?

Chat with the team behind TapCub — we usually reply within a few hours on business days.

What is smoke testing in simple terms?

A short check, run on every new build, that the most important things still work: the app starts, people can sign in, the core action and payment complete, key data arrives. If one fails, you stop and fix the build before any deeper testing or release.

What is the difference between smoke testing and sanity testing?

Smoke testing is broad and shallow, run on every new build to see whether it works at all. Sanity testing is narrow, run after a small change or fix to see whether that area behaves. Some sources use the two terms interchangeably.

What is the difference between smoke testing and regression testing?

A smoke test is a few minutes of checks on the critical paths and is a gate for the build. Regression testing is the large suite, often hours long, that checks that old features did not break. Run smoke first; if it fails, skip regression.

How often should you run a smoke test?

On every build or deploy, ideally automatically. Run a read-only subset on production right after each release, so a missing event or broken sign-in is found in minutes, not when a report looks wrong days later.

See this number on your own site

Add one script tag for web and apps, and the first reports fill in within minutes.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics