TapCub is live — analytics, insights and live chat, free on one platform
Product News

AI Onboarding: A Flow With AI Steps and Activation Before and After

AI onboarding puts AI at the steps where new users stall. A five-step flow, the activation metrics to judge it, and a worked before-and-after calculation with a confidence interval and the caveats.

Chen Yu Published Nov 20, 2026 11 min read Product News
Sample data: after the setup assistant and role-based checklist, project creation rose from 60.6% to 68.9% and activation from 32.0% to 36.2%.Sample data

AI onboarding means using AI at the specific steps where new users stall: answering setup questions, tailoring a checklist, helping with imports. Judge it by activation rate and step conversion, not by how clever the AI is. In the sample below, activation rose from 32.0% to 36.2% (95% interval +1.26 to +7.14 points), but a before-and-after number can mislead, so the caveats and a holdout design matter as much as the lift.

What you will learn

  • Put AI only where users stall and ask for help, then judge it by activation rate and the step it was meant to move
  • Activation rose from 32.0% to 36.2% (p = 0.0051, interval +1.26 to +7.14 points) but two different cohorts are not an experiment
  • A random 10% holdout removes traffic mix, seasonality and definition drift in one go; 2,000 sign-ups per period detects 4.2 points but not 3

What is AI onboarding, and where can AI help a new user?

AI onboarding means using AI at specific steps of the path from sign-up to a user's first real result: answering setup questions, tailoring a checklist, helping with data import. It is useful at the steps where new users stall and ask for help. It adds nothing at steps where they do not stall.

That restraint matters, because onboarding itself is a cost to the user. Nielsen Norman Group's research on app onboarding says it adds interaction cost and memory strain, advises avoiding it where possible, and recommends testing first: check whether users struggle without it, improve the design if they do, and keep an onboarding flow only if it measurably helps (NN/g). It also favours tips shown in context over a front-loaded tutorial. AI fits that preference when it answers the question a user has at the moment they have it.

AI stepWhat it doesWhere it helpsWhere it is wasted
Setup assistantAnswers "how do I connect X?" from your help materialSteps with a high question rateSteps nobody asks about
Role-based checklistPicks the next 3 tasks from the user's stated roleProducts with several distinct use casesProducts with one obvious first action
Data import helperSuggests how to map columns or fix errorsSteps with an upload or connectionAnywhere without messy input
Timed nudgesSuggests the next step when a user goes idleLong setups with a clear next actionShort setups; users who are simply thinking

A step that a user can already complete without help gains nothing from an AI layer. Find the stalls first, and only then decide where to put AI.

What should you measure to judge an onboarding flow?

Pick one outcome metric and a few diagnostic ones, and fix the definitions before you change anything.

MetricDefinitionWhy it matters
Activation rateUsers who reach the activation event within 7 days ÷ users who signed upThe outcome the flow exists to move
Step conversionUsers who complete step N ÷ users who completed step N−1Shows where people stall
Time to activateMedian days from sign-up to the activation eventSpeed to value
Setup questions per sign-upChats and tickets about setup ÷ sign-upsShows whether help is needed
Assisted shareSetup questions answered without a person ÷ setup questionsShows whether the AI step is doing work

Choose an activation event that reflects value, not effort. In the example below it is "viewed the first report with real data", not "created an account". For definitions of the other standard onboarding measures, see onboarding metrics, and for the surrounding process, customer onboarding system.

MetricDefinitionWhy it matters
Activation rateReached the activation event within 7 days ÷ sign-upsThe outcome the flow should move
Step conversionCompleted step N ÷ completed step N−1Shows where people stall
Time to activateMedian days from sign-up to activationSpeed to value
Setup questions per sign-upSetup chats and tickets ÷ sign-upsShows whether help is needed
Assisted shareSetup questions answered without a person ÷ setup questionsShows whether the AI step does work

Fix the definitions before you change anything; pick an activation event that reflects value, not effort.

What does an onboarding flow with AI steps look like?

Take a sample analytics product with a 5-step path. All numbers are sample data, recomputed in Python. Two cohorts of 2,000 sign-ups each: four weeks before the AI setup assistant launched, and four weeks after.

StepBefore: usersBefore: step rateAfter: usersAfter: step rate
1. Signed up2,0002,000
2. Verified email1,70085.0%1,71285.6%
3. Created a project1,03060.6%1,18068.9%
4. Connected a data source74071.8%88074.6%
5. Viewed the first report (activated)64086.5%72482.3%

Two AI steps were added: a setup assistant in the chat window that answers "how do I install this?" and "why is no data arriving?" from the help material, and a role-based checklist shown after email verification.

Read the table from the biggest stall. Before the change, only 60.6% of verified users created a project: that is where 670 people disappeared. After the change the step rate was 68.9%, +8.3 points, the largest improvement in the funnel. Step 4, connecting data, moved from 71.8% to 74.6%. That matches where the assistant answers questions, which is evidence that it did something.

The last step went the other way: 86.5% down to 82.3%. That is not a failure. More people now reach step 5, and the extra ones are marginal users who were less likely to finish, so the average step rate falls even though more people finish (724 against 640). Always read step rates together with counts.

Other sample measures over the same two periods: setup questions per sign-up fell from 540 ÷ 2,000 = 0.27 to 410 ÷ 2,000 = 0.205, 290 of the 410 (70.7%) were answered without a person, and the median time to activate fell from 3.1 to 2.4 days.

app.tapcub.com

Sign-up to activated, after the AI steps

2,000 sign-ups, 4 weeks

Signed up

2,000

Verified email

1,712

Created a project

1,180

Connected data

880

Activated

724

Before the AI steps 640 of 2,000 activated (32.0%); project creation was the largest stall at 60.6%.
Sample data
Sample data: after the setup assistant and role-based checklist, project creation rose from 60.6% to 68.9% and activation from 32.0% to 36.2%.

How do you calculate the before-and-after effect?

The activation rate is activated ÷ sign-ups:

  • Before: 640 ÷ 2,000 = 32.0%
  • After: 724 ÷ 2,000 = 36.2%
  • Difference: 4.2 percentage points, a relative lift of 13.1%.

To see whether the difference is larger than chance, use a two-proportion test:

  1. Pooled rate = (640 + 724) ÷ 4,000 = 34.1%.
  2. Standard error under "no difference" = √(0.341 × 0.659 × (1/2,000 + 1/2,000)) = 0.01499.
  3. z = 0.042 ÷ 0.01499 = 2.80, giving a two-sided p-value of 0.0051.
  4. 95% interval for the difference, with unpooled standard error 0.01498: 0.042 ± 1.96 × 0.01498 = +1.26 to +7.14 points, or +4.0% to +22.3% relative.

So the lift is statistically significant, and the plausible range runs from small to large. The size of the cohorts is also worth noticing: 2,000 sign-ups per period is just enough to detect a 4.2-point change at 80% power (the formula needs 1,999 per group). A 3-point change, from 32% to 35%, would need 3,884 per group, so this design could not have reliably detected a smaller improvement. For the formula, see A and B testing.

Why can a before-and-after number mislead?

Two different groups of people, in two different periods, with a change in between: that is not an experiment. Anything else that moved can produce the same difference. Researchers who run thousands of controlled experiments warn that "any figure that looks interesting or different is usually wrong" and that when a result looks too good, you should look for the bug before celebrating (Kohavi et al., 2014).

Check each of these before crediting the AI steps:

CheckWhat to compareIf it differs
Traffic source mixShare of sign-ups from each channel in the two periodsA campaign that brings more motivated users inflates activation
Plan or product changesRelease notes for both periodsA simpler sign-up form changes who signs up
CalendarSame weeks of the month; holidaysSeasonal users activate differently
Definition driftThe activation event and its trackingA tracking fix can move the number on its own
Sample ratioSign-up counts per period against expectationMissing data on one side

The stronger design is a holdout: show the AI steps to 90% of new users and withhold them from a random 10%, then compare activation between the two groups at the same time. That removes every item in the table at once. Your own product has to do the random assignment; the measurement afterwards is the same two-proportion test as above.

Which AI steps should you keep, and which should you cut?

Decide per step using the metric that step is meant to move.

ResultStep rate movesQuestions answered by AIDecision
Clear gainUp beyond the interval's lower boundManyKeep and extend
Gain at the right step, no gain elsewhereUp only where the AI answersManyKeep where it works; remove elsewhere
No changeFlatFewCut: NN/g's advice is to keep onboarding only if it measurably helps
Higher activation, lower day-7 returnUp, then downAnyLook at who the extra activations are before keeping it

In the example, the setup assistant and the checklist were added together, so the table above cannot tell you which one produced the project-creation gain. To separate them, switch one off for a random slice of users, or stage the launch a few weeks apart and compare step rates.

How do you measure AI onboarding in TapCub?

TapCub does not build product tours, in-app checklists or onboarding messages, and it does not run holdouts. Your product shows the steps; TapCub measures what happened. For the example:

  1. Recognise users. Call tapcub.login() at sign-in so each user appears in All-platform analytics › Users › People with a day-by-day timeline. Send the key steps as events: email verified, project created, data source connected, first report viewed.
  2. Build the funnel. In All-platform analytics › Behavior › Behavior analysis, add the five steps (up to 20 are allowed) and set the conversion window to 7 days. Each step shows count, rate against step one, rate against the previous step, drop-off and median time; the largest drop is marked.
  3. Compare the two periods. The funnel has no compare-period option, so run it once for the four weeks before launch and once for the four weeks after, and read the two step tables side by side. Use the breakdown on the first layer (for example by source) to check that the channel mix did not move.
  4. Check retention for the activated. In the same page, use retention to see whether people who activated in the later period come back in week 2 as often as before.
  5. Watch the questions. In Live chat › Reports, resolution rate and knowledge base hit rate show how many setup questions the AI answered from your knowledge base. TapCub's AI only drafts replies from that knowledge base; it does not change settings or perform setup for the user. Add the questions that stall people to Live chat › Knowledge base as exact-match Q&A, and pin them so visitors can read the answer without opening a conversation.

FAQ

What is AI onboarding?

Using AI at chosen steps between sign-up and a user's first result: answering setup questions, tailoring a checklist, helping with imports or nudging stalled users. It is justified only where users stall.

How do you measure whether AI onboarding works?

Compare the activation rate (activated within 7 days ÷ sign-ups) before and after, with a confidence interval, and check traffic mix and definitions. A random holdout of new users gives a cleaner answer.

What is a good activation rate?

It depends on your activation event and audience, so there is no reliable universal number. Compare yourself to your own history and watch the step where most users stall. The sample here went from 32.0% to 36.2%.

Does AI onboarding replace a human customer success team?

No. In the sample, 70.7% of setup questions were answered without a person and 29.3% still needed one. AI handles repeated questions; people handle unusual setups and judgement calls.

How many users do you need to test an onboarding change?

For a 32% baseline, about 1,999 per group detects a 4.2-point lift with 80% power, and 3,884 per group detects a 3-point lift. Smaller changes need many more.

Sources

Filed underProduct News

Was this article helpful?

We use your answers to fix and expand this page.

Chen Yu

Technical Engineer

Focuses on tracking code, SDKs, and API integrations, with practical guidance on implementation checks and troubleshooting.

Measure activation step by step

Build the funnel with a 7-day window, break it down by source and see where the AI-answered setup questions land in chat reports.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics