TapCub is live — analytics, insights and live chat, free on one platform
Product News

Product benchmarking: compare your usage metrics the right way

Benchmark how people use your product, not only rival features. Build a band from your own 12 weeks, test any outside number for definition and unit, and avoid a mix shift that fakes an improvement.

Lin Zhou Published Nov 21, 2026 11 min read Product News
Sample data: the audience moved from 40% to 60% desktop, so the overall activation rate rose from 40.0% to 44.0% while mobile and desktop each fell by a point.Sample data

Product benchmarking compares your product's measurements with a reference point; for usage metrics the reliable one is your own history. Build a P25 to P75 band from your last 12 weeks, judge new values against it, and use an outside benchmark only when its definition, unit and period match yours.

What you will learn

  • An outside benchmark is usable only when definition, unit, period and method match; until then your own history is the better benchmark
  • In the sample, week-12 activation of 44.6% is above the P25 to P75 band of 39.55 to 41.80, but first rule out tracking changes and mix shifts
  • A mix shift can raise the overall rate by four points while every segment falls: 40.0% to 44.0% with both segments a point worse

What is product benchmarking, and which kind do you need?

Product benchmarking is comparing your product's measurements against a reference point to see whether a number is good, bad or ordinary. Most guides treat it as a feature comparison with rivals, which the competitor benchmarking guide covers. This guide is about the other kind: benchmarking how people use your product, such as activation, retention and feature adoption, because that is where the reference points are easiest to get wrong. The general idea is defined on the benchmarking page.

There are three kinds of reference point, with very different reliability:

ReferenceCompares you withReliabilityMain risk
InternalYour own past weeks and your own segmentsHigh: same definition, same productSays nothing about how you compare with the market
PublishedA public dataset with a stated methodMedium: definitions are visibleThe unit or population differs from yours
Peer exchangeNumbers other companies share privatelyLow unless definitions are writtenSelf-selected, unaudited, definitions unknown

The claim to hold on to is that an outside benchmark is only usable once its definition, unit and period match yours; until then your own history is the better benchmark. The rest of the guide shows how to build the internal one properly, then how to test an outside number before you rely on it.

Which usage metrics are worth benchmarking?

Pick metrics that describe whether people get value, with a written definition each, so that you can compare like with like later.

MetricDefinitionBetter whenWatch out for
Activation rateNew users who do the key first action within 7 days ÷ new usersHigherCounting a page view as activation
Week-1 retentionNew users active again in days 1 to 7 ÷ new usersHigherMixing new and returning users
Feature adoptionActive users who used the feature in the period ÷ active usersHigher, if the feature mattersAdoption that does not change retention
Task completion timeMedian time to finish an ordered taskLowerUsing the mean, which outliers distort
Error rateErrors ÷ page viewsLowerCounting bots as users
Support contactsConversations ÷ active usersLowerFewer contacts because help is hard to find

Keep to four to six. A benchmark table of twenty metrics is read once and never again. Compute every rate on real people only, since crawlers and bots in the denominator drag every rate down.

How do you benchmark against your own history?

Take the last 12 weeks of a metric, sort them and read the quartiles. The 25th percentile (P25) is the value below which a quarter of the weeks fall, the 50th is the median and the 75th (P75) is the value below which three quarters fall. The band between P25 and P75 is what a normal week looks like for you.

Compute the band from the weeks before the one you are judging, so that the new value does not move its own yardstick.

Sample data: weekly values for three metrics over 12 weeks. The band uses weeks 1 to 11 and the judged value is week 12.

MetricP25MedianP75Week 12Position
Activation rate (%)39.5540.9041.8044.60Above the band
Week-1 retention (%)22.0522.5022.9522.10Inside, near the bottom edge
Feature adoption (%)33.0034.0036.0039.00Above the band

The quartiles use the inclusive method (Python's statistics.quantiles(values, n=4, method='inclusive')). Three rules turn the table into decisions:

  • Inside the band: normal variation; do nothing on this metric.
  • Below P25: a problem to investigate this week.
  • Above P75: do not celebrate yet. First rule out a tracking change, a one-off campaign and a shift in who is visiting; the next section shows how a shift can create an improvement that does not exist.

In the sample, activation and feature adoption are above the band and week-1 retention sits just inside it. A team that cheers for activation while retention has not moved should check whether the new users are a different kind of user.

MetricP25MedianP75Week 12
Activation rate (%)39.5540.9041.8044.60
Week-1 retention (%)22.0522.5022.9522.10
Feature adoption (%)33.0034.0036.0039.00

Sample data: bands from weeks 1 to 11 with the inclusive quartile method; week 12 is the value being judged.

When is an outside benchmark usable?

Before you compare with a published number, check four things.

  1. Same definition. Is "active" or "retained" defined as you define it?
  2. Same unit. Is the number a share of page loads, of sessions, of users or of whole websites?
  3. Same period and population. Which dates, which countries, which devices?
  4. Stated method and date. Can you find how the number was produced, and when?

Here is a case that fails the second test. Google's Core Web Vitals say a page passes when its Largest Contentful Paint is within 2.5 seconds, Interaction to Next Paint is 200 milliseconds or less and Cumulative Layout Shift is 0.1 or less, each measured at the 75th percentile of page loads. The HTTP Archive's Web Almanac 2024 reports that 43% of mobile and 54% of desktop websites in its dataset passed all three in 2024. Suppose your own monitoring shows 66% of mobile page loads meeting the thresholds. That is not "better than the 43%": your 66% is a share of page loads, while the 43% is a share of websites, where a site either passes at the 75th percentile or does not. The comparable figure from your side is one yes or no for your whole site, and the honest statement is "our mobile site passes at the 75th percentile, as 43% of websites in the dataset do".

The same test applies to any retention or conversion benchmark quoted in a report: ask what the denominator is, because a share of carts and a share of checkouts started are different numbers even when both are called abandonment.

How do you stop a mix shift from fooling you?

An overall metric can move for a reason that has nothing to do with the product: the mix of people changed.

Sample data: a product has mobile and desktop users with different activation rates.

WeekMobile shareMobile rateDesktop shareDesktop rateOverall
A60%30%40%55%0.6 × 30 + 0.4 × 55 = 40.0%
B40%29%60%54%0.4 × 29 + 0.6 × 54 = 44.0%

The overall rate rises four points, from 40.0% to 44.0%, while both segments got a point worse. The product did not improve; the audience moved toward desktop users, who activate more often. Compare the same segment week to week, and when you report an overall number, report the mix beside it. Three other traps:

  • Seasonality. Compare the same weekday and, for annual patterns, the same period last year, not the previous month.
  • Small samples. A weekly rate based on 40 people can swing by ten points. Pool weeks or lower the granularity until each point has at least a few hundred people.
  • Definition drift. If the tracking changed in week 8, the band before and after is not one series. Mark the date and rebuild the band.

How unstable a small sample is can be calculated: the 95% margin of error of a rate is about 1.96 × √(p × (1 − p) ÷ n). When the true activation rate is 40%, a week of 40 people has a margin of about ±15.2 points and a week of 400 people about ±4.8 points. A ten-point swing in a 40-person week can therefore be noise alone, while the same swing in a 400-person week deserves a look.

app.tapcub.com

Overall rate up, every segment down

sample activation rate
Overall, week A40.0%
Overall, week B44.0%
Mobile, week A30.0%
Mobile, week B29.0%
Desktop, week A55.0%
Desktop, week B54.0%
Sample data
Sample data: the audience moved from 40% to 60% desktop, so the overall activation rate rose from 40.0% to 44.0% while mobile and desktop each fell by a point.

How do you set a target from a benchmark?

Use the band to set a target you can defend. For a metric below P25, aim first for the median, since that has been normal for you. For a metric already inside the band, choose a target at P75, which you have already reached in a quarter of recent weeks. A target far above anything in your history needs a hypothesis and a plan, not just a number.

Sample data: week-1 retention has a median of 22.50% and a P75 of 22.95%. A target of 25% would be well above every one of the 12 weeks (the best was 23.4%), so it belongs in a project with a hypothesis, not in a dashboard goal. A target of 23.0% is a realistic first step.

How do you do this in TapCub?

TapCub supplies your side of every comparison above. It does not provide a dataset of other companies' product metrics, so the outside half comes from published sources checked with the four tests.

  1. Save each metric once. In All-platform analytics › Behavior › Metrics & SQL, define each rate as a saved metric (an event plus an aggregation such as unique people, or a formula), so the definition lives in one place and every chart uses it.
  2. Read the weekly series. In All-platform analytics › Behavior › Behavior analysis, chart the metric by week for the last 12 weeks, and use the table to read the values for the quartile calculation. The retention report gives week-1 return by cohort, and the lifecycle view separates new, returning, resurrecting and dormant users so you do not mix them.
  3. Split by segment. Break the same chart down by device or source, and compare against the previous period or the same period last year, optionally aligned by weekday, to check for the mix shift above.
  4. Mark tracking changes. Add every release or tracking change in Configuration › Annotations, so the series is read in sections, not as one line.
  5. Compare Core Web Vitals correctly. Monitoring › Performance shows real-user vitals for your own site; read the 75th percentile values, not the share of loads, before you compare with a published figure.

FAQ

What is product benchmarking?

Comparing your product's measurements with a reference point, such as your own history, a published dataset or a rival's features. This guide covers usage metrics: activation, retention, feature adoption, task time, errors and support contacts.

How do you benchmark a product?

Choose four to six metrics with written definitions, build a band from your last 12 weeks (P25 to P75), judge new values against it and test any outside benchmark for definition, unit, period and method before comparing.

What is a good benchmark for product metrics?

Your own history under a fixed definition is the most reliable one. Published figures help only if the unit and definition match yours; for example, the Web Almanac's 43% of mobile websites passing Core Web Vitals is a share of websites, not of page loads.

How often should you benchmark?

Weekly for the series and the band, quarterly for target-setting and outside comparisons. Rebuild the band whenever tracking, definitions or the product's main audience change.

Sources

  • HTTP Archive, Web Almanac 2024: Performance — Core Web Vitals pass rates, 43% of mobile and 54% of desktop websites in 2024. Checked 2026-11-21.
  • web.dev, Web Vitals — the good thresholds and the 75th percentile rule. Checked 2026-11-21.
  • Python documentation, statistics.quantiles — the inclusive method used for the quartiles. Checked 2026-11-21.
Filed underProduct News

Was this article helpful?

We use your answers to fix and expand this page.

Lin Zhou

Product Experience Specialist

Writes about product experience, practical workflows, and growth, turning complex processes into clear steps for everyday decisions.

Keep your own benchmark in one project

Saved metrics, weekly series, retention by cohort, segment breakdowns, release annotations and real-user page speed in one place.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics