What is diagnostic analytics? Methods and a worked example

What is diagnostic analytics?

Diagnostic analytics is the type of analysis that answers why did this happen? It takes a change in a metric (sign-ups down 18%, churn up two points) and breaks it down by segment, time and related events until the cause is narrow enough to act on.

It is usually described as the second of four types of analytics:

More on all four in types of data analytics.

Techniques

• Drill-down and breakdowns: split the metric by channel, country, device, page, plan, version.
• Contribution analysis: how much of the total change each segment explains.
• Metric trees: decompose the metric into its drivers (sign-ups = visitors × sign-up rate) and see which driver moved.
• Segmentation and cohorts: did new users change, or existing ones?
• Comparison with a control period: same weekday last week, same period last year.
• Anomaly detection: flag which segments moved more than usual.
• Root-cause methods: the "5 whys" from the Toyota Production System, asked of data rather than of people.

A six-step method

1. Confirm the change is real. Check tracking changes, deploys, bot traffic and data delays before analysing behaviour. Many "drops" are instrumentation.
2. Size it against normal variation. Is −18% outside the usual weekly range? Compare with the same weekday and with seasonality.
3. Decompose the metric. Is it fewer visitors, a lower conversion rate, or both?
4. Break down and rank contributions. Find where the change is concentrated.
5. Form and test hypotheses. Look for an event that explains the concentrated segment: a release, a campaign change, a competitor, an outage. Check it with other data, including support chats.
6. Act and verify. Fix, then watch the same breakdown to see the segment recover.

A worked example: contribution analysis

Sample data: weekly conversions fell from 2,000 to 1,640 (−360, −18%). Breakdown by channel:

Contribution = channel change ÷ total change; for paid search, −290 ÷ −360 = 80.6%. Email grew, so its contribution is negative: it offset part of the drop. Drilling into paid search by campaign shows one campaign went from 240 conversions to 0 on Tuesday: its budget had run out. Restarting it explains most of the drop; the smaller organic and direct declines are within normal weekly variation and need no action yet.

Pitfalls

• Mix shift. Every segment's rate can rise while the total falls, because traffic moved to a lower-converting segment. This is Simpson's paradox; check both rates and volumes.
• Slicing until something appears. With enough dimensions, some segment will look unusual by chance. Prefer dimensions you would have checked anyway, and treat surprises as hypotheses.
• Stopping at the first explanation. A paused campaign may explain 80%; the other 20% might be the start of a real trend.
• Correlation taken as cause. Two lines moving together after a release do not prove the release caused it.

Diagnostic analytics in TapCub

• All-platform analytics › Operations › AI Assistant › Why did it change explains a metric change against the previous period or the same period last year. It splits by channel, referrer domain, country, device, entry page, platform and related dimensions, one or two levels deep, and ranks each value by contribution (that value's change ÷ the total change), the same calculation as the table above. See ask the AI assistant.
• Monitors with a forecast band or anomaly condition flag unusual moves so you start diagnosing early, and annotations record releases and campaigns that step 5 asks about.
• For traffic drops specifically, the checklist in sudden drop in website traffic covers tracking and SEO causes.

Sources

• Davenport, T. H. and Harris, J. G. (2007). Competing on Analytics: The New Science of Winning. Harvard Business School Press.
• Ohno, T. (1988). Toyota Production System: Beyond Large-Scale Production. Productivity Press.
• Pearl, J. and Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect. Basic Books.