TapCub is live — analytics, insights and live chat, free on one platform
Analytics

Mobile Testing Tools: Four Kinds and What Each One Tells You

Mobile testing tools answer four different questions: does it work, does it work on their devices, is it fast and stable, and what do real users hit after release. A before-and-after checklist and a worked release comparison.

Xu An Published Nov 19, 2026 13 min read Analytics
Sample data: the new version looks 1.0 point worse overall, but one model at 15.0% crash users explains it; every other model is better than the old version.Sample data

Mobile testing tools are best sorted by the question they answer: does it work, does it work on their devices, is it fast and stable, and what do real users hit after release. The first three run before a version ships; the fourth, real-user data, is the only one that sees what actually happened. Below: the four kinds, a worked release comparison and a before-and-after checklist.

What you will learn

  • Pick tools by the question they answer, not from a ranked list: function, compatibility, performance, real users
  • Choose test devices from your own active-user shares: six models covered 81% and eight covered 90% in the sample
  • Compare releases by crash rate per version and per device; a 2.0% aggregate hid one model at 15.0%

Which kind of mobile testing tool answers which question?

Lists of mobile testing tools mix products that answer very different questions. Sort them by the question instead. Android's own testing guide splits tests by subject into functional, performance, accessibility and compatibility; add what real users do after release, and you have four questions that need four kinds of tool.

QuestionKind of toolWhen it runsWhat it cannot tell you
Does it do what it should?Unit, integration and UI test frameworks, run locally or on a deviceEvery commit and before releaseWhether anyone uses the feature, or whether it works on a phone you did not test
Does it work on their devices?Emulators, device labs and beta channelsBefore release, on a device matrixHow many of your users are on each device
Is it fast and stable?Profilers, load and startup benchmarksBefore release and on release candidatesBehaviour on real networks, real battery levels and real data
What do real users do and hit?App analytics: crashes, errors, versions, events, funnelsAfter release, continuouslyWhy a bug happens; it shows where and for whom

The first three kinds are the ones most lists cover. They run before the version reaches users. The fourth kind is the only one that sees what happened after release, and it is the one that tells you whether the first three were enough. This guide takes them in order, then gives a before-and-after checklist and a worked release comparison. If your question is whether people can use the app, that is usability testing, covered in mobile app usability testing.

Functional testing: does the app do what it should?

Functional tests check behaviour: tap Continue, expect the Welcome screen. Android's testing fundamentals describe three sizes. A small (unit) test checks a method or class, a medium test checks two or more units together, and a big (end-to-end) test checks a whole screen or user flow. They also distinguish where tests run: local tests run on your development machine, instrumented tests run on a physical or emulated device.

Three decision rules keep this layer cheap:

  • If a rule can be tested without a screen, test it without a screen. Pricing, validation and state logic belong in small local tests. They run in seconds and rarely flake.
  • Keep the big device tests for the money paths. Sign-up, login, checkout and the one feature you sell. Android's guide notes that code which cannot be isolated forces bigger, slower, flakier tests, which is the argument for testable structure.
  • Automate what you repeat; explore what is new. Manual testing scales poorly and misses regressions, as the same guide says, but a new feature still needs a person trying odd inputs once before you write the script.

What this layer cannot do is tell you which flows matter. That comes from usage data. If 2% of sessions ever reach a settings screen, a long device test for it is lower priority than a test for the flow 60% of sessions use.

Compatibility testing: which devices are worth testing on?

You cannot test every phone, so choose the matrix from your own users, not from a generic list. Rank device models and OS versions by active users and add rows until you cover the share you want.

Sample data: a store app's active users by device model over 30 days.

ModelShare of active usersCumulative
Model A22%22%
Model B18%40%
Model C14%54%
Model D11%65%
Model E9%74%
Model F7%81%
Model G5%86%
Model H4%90%
Everything else10%100%

Six models cover 81% of users and eight cover 90%. The last 10% is a long tail of many models, none above about 1% in this sample, so extra devices add little. Two rules follow:

  • Start with the models that cover about 80–90% of users, then add risky ones. Risky means oldest supported OS version, smallest screen, lowest memory, plus any model with a crash history.
  • Add by evidence. When a crash cluster shows up on one model in production, add that model to the matrix and keep it there.

OS version works the same way: list versions by active users, test the newest, the oldest you support, and the most common in between. Whether your analytics can report device model, OS version, resolution and network depends on what the SDK collects, so check that the fields arrive before relying on them.

Performance testing: is it fast and stable enough?

Performance tests check speed, memory and responsiveness before users feel them. Pick budgets before you test, otherwise every result is "looks fine".

  • Startup time. Measure cold start (the process starts from nothing) separately from hot start, on a low-end device, not your development phone.
  • Responsiveness. On Android, an app not responding (ANR) is triggered when it has not responded to an input event within 5 seconds, so any main-thread work near that range is a defect, and much shorter freezes are already visible to users.
  • Memory. Low-memory devices are where out-of-memory crashes appear; Android's crash guide suggests testing with a low-memory emulator when you see them.
  • Network. Users move in and out of coverage. The same guide says network exceptions are normal operating conditions, so test airplane mode and slow links, and check the app degrades instead of crashing.

For a reference point on what counts as bad, Google Play publishes thresholds for its own Android vitals: a user-perceived crash rate of 1.09% of daily users overall, or 8% on a single phone model, and a user-perceived ANR rate of 0.47% overall, or 8% on a single model. Those are measured over 28 days and use Google's own definitions, so treat them as context for your numbers, not as a target your analytics will match.

Lab performance testing has a blind spot: it uses your devices, your network and your data. A release can pass every benchmark and still be slow for the 8% of users on one model with a full storage drive. That gap is what the fourth kind closes.

Real-user data: what happened after release?

After release, the question changes from "does it work?" to "how many people hit a problem, on which version and device?". Counts mislead here because a new version starts with few users. Use rates: crash users divided by active users, per version.

Sample data: version 4.3 is rolled out gradually. After three days it has 6,000 active users; version 4.2 has 40,000 over the same week.

SliceActive usersCrash usersCrash-free share
Version 4.2, all devices40,00040099.0%
Version 4.3, all devices6,00012098.0%
Version 4.3, Model X6009085.0%
Version 4.3, all other models5,4003099.4%

The crash-free share is 1 − crash users ÷ active users. For 4.3: 120 ÷ 6,000 = 2.0%, so 98.0% crash-free, against 1.0% for 4.2. The gap is 1.0 percentage point. A standard two-proportion check gives a standard error of 0.19 points, so the 95% interval for the difference is about 0.63 to 1.37 points: this is not noise.

Now slice by device. Model X has 600 users and 90 crash users, or 15.0%. Every other model together has 30 crash users out of 5,400, or 0.56%, which is better than 4.2's 1.0%. The release is not broadly worse; one model is badly broken. The fix is a hotfix or a staged-rollout halt for that model, not a rollback of everything. The aggregate number alone would have sent you to the wrong decision.

Startup time reads the same way: if the median cold start moves from 1.8 s to 2.6 s, that is 44.4% slower, and you check whether it is everywhere or on older devices only.

What to check before and after each release

WhenCheckPass rule (example)
BeforeSmall and medium tests on every commitAll green on the release branch
BeforeDevice tests on the money pathsPass on the matrix covering about 80–90% of users
BeforeStartup and memory on the lowest supported deviceWithin the budget you set
BeforeAnalytics events fire with required propertiesEach planned event arrives with its required fields
Rollout day 1–3Crash-free share by version and by deviceNew version within about 0.5 point of the old one overall; no single model far worse
Rollout day 1–3Custom errors and funnel steps for the new flowsNo step has dropped against the previous version
Day 7Retention and median startup versus previous versionNo unexplained drop
Any timeOne model or OS version far worse than the restHalt the rollout for that slice, then fix

Treat the 0.5 point in the pass rule as a starting rule of thumb you tune to your own history, not a standard. What matters is that you wrote the rule before the release.

Common mistakes in choosing and using mobile testing tools

  • Treating a long list of tools as a coverage plan. Coverage is about questions answered, not products used. Check each of the four questions has an owner.
  • Reading crash counts instead of crash rates. The old version always has more crashes because it has more users.
  • Averaging across devices. The worked example shows a 2.0% aggregate hiding one model at 15.0%. Always slice by device and OS.
  • Testing on the newest phones. The lab has the fast phone; your users do not all have one.
  • Skipping analytics validation. If a new screen sends no events, the funnel looks like a drop. Verify events before blaming users.
  • Expecting analytics to replace tests. It tells you something broke after users met it; tests stop it before they do.

How to do the post-release half in TapCub

TapCub is not an automated testing tool: it does not run test scripts, rent devices or profile code. It covers the fourth question and part of the third, once real users have the build. Here is the 4.3 rollout above, in the console:

  1. Before release, plan the events. In All-platform analytics › Data › Tracking plan, list the events the new flow should send, with required properties and status. After a test build runs, switch on real-time validation (last 24 hours) to see whether each event arrived and whether its required properties are complete.
  2. On rollout day 1, open the version table. App analytics › Users › Versions lists each version with its share of active users, crash count and first-seen date. TapCub also adds a release marker on trend charts the first day a version appears, so later changes line up with it.
  3. Read the crash-free share. App analytics › Quality & attribution › Stability shows the crash-free share (1 − crash users ÷ active users), ANRs, UI freezes, starts and median cold start, hot start and memory. Use the header filter to narrow to 4.3, then compare with 4.2.
  4. Slice by device. App analytics › Quality & attribution › Devices breaks users down by model, OS version and more, ordered by active users. Combine it with the Stability filter to find the model that crashes. Which device fields appear depends on what each SDK collects.
  5. Handle errors in groups. Error groups can be marked active, resolved or ignored. Business errors your app sends as custom errors are counted on their own tab and do not change the crash-free share.

TapCub does not stop a rollout for you or fix anything; it shows where and for whom. Pair it with the release controls of your store and your own pipeline. For wider app metrics, see mobile app KPIs; for versions and release tracking, see app versioning and improve app performance. For what lab-style probing of a live site looks like, see synthetic monitoring.

FAQ

What are mobile testing tools?

They are software that checks a mobile app before and after release. In practice they fall into four kinds: functional test frameworks, compatibility labs and emulators, performance profilers and benchmarks, and real-user analytics for crashes, versions and events. Each answers a different question, so one product rarely covers all four.

Which mobile testing tools do I need first?

Start with small automated tests for your core logic, one device test per money path, and crash reporting with version and device filters. That covers the two questions that hurt most: did we break something, and did a release break it for real users. Add device labs and performance benchmarks as the app and audience grow.

How many devices should I test on?

Enough to cover most of your actual users. In the sample above, six models covered 81% and eight covered 90%. Rank your own device models by active users, stop where extra devices add little, then add the oldest supported OS and any model with a crash history.

What is a good crash-free rate?

There is no single number. Google Play flags apps with a user-perceived crash rate above 1.09% of daily users overall or 8% on one model, but definitions differ between tools. Compare each release with your previous release and watch for any single model far worse than the rest.

Is manual testing still needed?

Yes, for new features, usability, and odd inputs. Android's guidance says manual testing scales poorly and misses regressions, so automate what you repeat. Use people for exploration and judgment, and use real-user data to confirm that what passed in the lab also works in the field.

Sources

Filed underAnalytics

Was this article helpful?

We use your answers to fix and expand this page.

Xu An

Customer Experience Consultant

Writes about live chat, visitor conversations, and self-service, sharing practical ways to make customer communication clearer and more effective.

Watch the post-release half in TapCub

See crash-free share by version, device mix and release markers for your app. TapCub does not run test scripts; it shows what real users hit.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics