TapCub is live — analytics, insights and live chat, free on one platform
Analytics

Data Pipeline Monitoring: Freshness, Volume and Schema Checks

Data pipeline monitoring needs a few checks with real thresholds. Learn freshness, same-weekday volume baselines, schema completeness and duplicate rules, with a worked bad week and a backtest step.

Xu An Published Nov 21, 2026 11 min read Analytics
Sample data: Wednesday reached 5,400 of 10,275 expected events (52.6%) and Sunday received none. The weekday baseline flags both without flagging the normal weekend dip.Sample data

Data pipeline monitoring is a short set of automatic checks that tell you when data is late, the wrong size, the wrong shape or the wrong value: freshness, volume against a same-weekday baseline, completeness, duplicates and ranges. Run status alone is not enough. This guide gives starting thresholds, a worked bad week and a backtest step.

What you will learn

  • Five checks cover most incidents: freshness, volume, completeness, duplicates and value ranges; each gets a warn and an alert level
  • Compare volume with the median of the same weekday: a flat 80% rule alerts every weekend in the sample, a weekday baseline does not
  • Backtest each rule over 28 days before enabling it and give every alert an owner and a first step

What a pipeline can get wrong

Data pipeline monitoring is a set of automatic checks that tell you when data arrives late, in the wrong amount, in the wrong shape or with wrong values, before someone notices a broken dashboard. A job that finishes with a green "success" status can still deliver nothing useful, so run status alone is not monitoring.

Five failure modes cover most incidents:

FailureWhat you seeCheck that catches it
LateYesterday's numbers are missing at 9 amFreshness
Missing or surgingVolume far below or above normalVolume against a baseline
Wrong shapeA required field is empty or a type changedSchema and completeness
DuplicatedCounts jump, totals doubleDuplicate rate on a unique ID
Wrong valuesNegative revenue, dates in the futureRange and sanity rules

The next sections give each check a starting threshold. Treat the numbers as defaults to tune, not as standards.

Freshness: how old is the newest data

Freshness is the time since the most recent record was loaded. It is the cheapest check and catches the most common failure, a job that stopped. Google Cloud's guidance on pipeline metrics describes the same idea as the age of the most recent fully processed item, and treats it as a latency signal.

Set two levels per feed, based on how often the feed should update. The dbt documentation's own example configures a source to warn after 12 hours and fail after 24 hours; that suits a feed that should update daily. A reasonable starting rule:

Expected updateWarn afterError after
Every 5 minutes15 minutes30 minutes
Hourly2 hours4 hours
Daily26 hours36 hours

The rule: warn at about twice the expected gap and error at about four times, with a little extra for a daily feed so that a slow job does not page anyone at 6 am. Measure from an event or load timestamp, not from the time the check runs.

Volume: compare with the same weekday

Volume checks compare today's record count with what is normal for that day. A flat threshold fails because most data has a weekly pattern. Sample data: four weeks of daily event counts give these same-weekday medians.

DayMedian of last 4 weeksWarn below 80%Alert below 60%
Mon10,2508,2006,150
Tue10,3508,2806,210
Wed10,2758,2206,165
Thu10,2508,2006,150
Fri9,9757,9805,985
Sat7,2755,8204,365
Sun7,1505,7204,290

A flat rule shows why this matters. The mean of the seven medians is 65,525 ÷ 7 = 9,361 a day. A flat "alert below 80% of average" threshold is 7,489, which Saturday (7,275) and Sunday (7,150) fall under every week. Same-weekday medians give weekends their own baseline and remove those false alarms.

Use the median, not the mean, so that one past outage does not drag the baseline down. Add an upper bound as well, at 150% for example, because a surge can mean duplicates or a bot.

DayMedian of last 4 weeksWarn below 80%Alert below 60%
Mon10,2508,2006,150
Wed10,2758,2206,165
Sat7,2755,8204,365
Sun7,1505,7204,290

Same-weekday median of four weeks; add an upper bound near 150% to catch surges and duplicates.

Reading a bad week

Sample data: a fifth week is compared with the table above.

DayEvents% of baselineStatus
Mon10,280100.3%OK
Tue10,33099.8%OK
Wed5,40052.6%Alert
Thu10,15099.0%OK
Fri9,89099.1%OK
Sat7,280100.1%OK
Sun00%Alert

Wednesday at 52.6% is below the 60% line, so someone is told the same day. Sunday at zero is a stopped pipeline; a freshness check on an hourly feed would have caught it within 4 hours instead of waiting for the daily total. The two alerts have different causes (a partial outage and a full stop), and the table is how you tell them apart. A volume check on a daily total is a backstop, not the fastest alarm.

app.tapcub.com

Wednesday

52.6%

▼ below 60% alert line

Sunday

0

▼ pipeline stopped

Other days

约等于基线

within normal range

Daily events: baseline vs this week

Baseline (4-week median)This week
MonTueWedThuFriSatSun
Sample data
Sample data: Wednesday reached 5,400 of 10,275 expected events (52.6%) and Sunday received none. The weekday baseline flags both without flagging the normal weekend dip.

Schema and completeness: is the shape still right

Schema checks protect against silent breakage after a release. Three checks cover most cases:

  • Required fields present. For each important event, the share of rows where the required properties are filled. Alert if it drops below 99% for fields used in revenue or conversion numbers.
  • Type stability. A field that was a number and now arrives as text breaks sums without raising an error.
  • New or missing events. An event name that appears for the first time, or one that stops, is usually a release you did not know about.

Sample data: 2,000 order events arrive in a day and 1,940 have a revenue value. Completeness is 1,940 ÷ 2,000 = 97.0%. Against a 99% line that is an alert, and 60 orders are missing revenue. Fix the cause in the sending code, then decide whether to backfill.

Duplicates and value ranges

A duplicate check counts repeated unique IDs. Sample data: 10,300 events in a day with 103 repeated event IDs is 103 ÷ 10,300 = 1.0%. A starting rule is warn above 0.5% and alert above 2%, since a retry loop or double-firing tag can multiply counts quickly.

Range rules catch values that are possible but wrong: negative prices, an order total of zero, timestamps in the future or more than a day old, and country codes outside a list. Write five rules for the fields that feed your most important metric and add one only when an incident shows a gap.

Lag and errors: the other two signals

Google's SRE guidance describes four golden signals for services: latency, traffic, errors and saturation. For a pipeline they translate directly. Freshness is latency, volume is traffic, failed records and job errors are errors, and a growing backlog of unprocessed items is saturation.

Track error rate as failed records divided by attempted records. A starting rule is warn at 1% and alert at 5% for a feed where a failed record is lost. For queue-based pipelines, alert when the oldest unprocessed item is older than your freshness error level, because that is the earliest sign of a backlog.

Who gets told, and how

The SRE book advises that every page should be urgent and actionable, and that subcritical problems belong on dashboards or in tickets rather than in noisy email. Apply that with two tiers:

LevelMeaningWhere it goes
WarnWorth looking at todayTeam channel or ticket
AlertReports are wrong nowNamed owner, immediate notification

Every alert needs an owner and a first step written next to the rule. An alert with no owner is a notification nobody acts on.

Tune thresholds with a backtest

Before enabling a rule, replay it over the last 28 days. Count how many times it would have fired and compare with the incidents you actually know about. If it fires more than once a week with no real incident, loosen it; if a known incident did not fire, tighten it. Sample data: the 80% warn line over the four sample weeks fires zero times, and the 60% alert line also zero, which is the result you want from a quiet month.

Review every alert that fired in the past month and write one of three labels on it: real, false or unclear. Retire or change any rule where more than half the alerts were false.

What to do when an alert fires

A short runbook turns an alert into a fix. Write these four steps next to every alert rule:

  1. Confirm it is real. Check the source system for the same period. If the source also shows a drop, the pipeline is fine and the business changed; if the source is normal, the pipeline lost data.
  2. Find where it stopped. Walk the chain in order: sender, collection endpoint, queue, storage, reports. The first stage whose count is lower than the one before it is where data is lost.
  3. Fix and mark it. Repair the cause, then add an annotation with the date range so nobody reads the gap as a business trend.
  4. Decide on backfill. If the lost events can be replayed, replay them once and recheck the counts; if not, record the gap as permanent.

In TapCub

TapCub does not monitor your warehouse or your ETL jobs; it monitors the events arriving into TapCub, which is the first step of the pipeline for web, app and server data.

  1. In All-platform analytics › Data › Tracking plan, record the events you intend to collect with their required properties, then turn on live verification (last 24 hours). It shows the arrivals per event and how complete the required properties are, which is the schema and completeness check above.
  2. New event names that you did not plan appear in All-platform analytics › Data › Event inbox until someone claims them, which catches the "release you did not know about" case.
  3. In All-platform analytics › Operations › Monitors, create a check on a built-in metric such as page views, sessions or error rate, or on a saved custom metric such as the daily count of one event. Pick hourly, daily, weekly or monthly, set the condition, and use the dry run: it computes without sending and tells you how many times the rule would have fired in the past 30 days, which is the backtest.
  4. Alerts go to email, WeCom, DingTalk, Feishu or a webhook. A monitor only notifies; it does not change anything. The same alert is not repeated for about 60 minutes by default.

FAQ

What should I monitor in a data pipeline?

Start with five checks: freshness (how old the newest data is), volume against the same weekday, required-field completeness, duplicate rate and value ranges. Add error rate and backlog for queue-based pipelines. Each gets a warn and an alert level.

How do I set alert thresholds for data volume?

Compare each day with the median of the same weekday over the last four weeks, and alert below 60% and warn below 80%. Add an upper bound near 150%. Backtest the rule over 28 days before turning it on.

What is data freshness and how do I measure it?

Freshness is the time since the newest record was loaded, measured from an event or load timestamp. A starting rule is to warn at about twice the expected update gap and error at about four times.

How do I reduce false alerts?

Use same-weekday baselines instead of flat thresholds, use medians, backtest every rule, and review each month's alerts as real or false. Retire rules where most alerts are false, and keep a named owner on every alert.

Is run status enough to know a pipeline works?

No. A job can finish successfully and load nothing, half the data or the wrong fields. Pair run status with freshness, volume and schema checks on the data itself.

Sources

Filed underAnalytics

Was this article helpful?

We use your answers to fix and expand this page.

Xu An

Customer Experience Consultant

Writes about live chat, visitor conversations, and self-service, sharing practical ways to make customer communication clearer and more effective.

Watch the data as it arrives

TapCub checks event arrivals and required-property completeness against your tracking plan, and monitors can alert on volume or error rate.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics