Try it on your own data
Every report in this article is in the free plan. One snippet for web and apps, up to 10 sites.
Start freeTo choose a chatbot provider, weight your needs so they add to 100, score each shortlisted provider 1 to 5 on the same criteria, compute Σ(weight × score) ÷ 100, and check whether the winner survives a 10-point shift in weights. In the example, the same three providers rank Y, X, Z for a small store and Z, Y, X for a regulated EU service; no provider wins for everyone, only for a set of weights.
What you will learn
- Weights express your needs: the same scores gave 3.75 for provider Y with buyer A weights and 3.45 with buyer B weights, while Z moved from 2.85 to 3.90
- If a 10-point weight shift changes the winner (a 0.45 lead became 0.05), treat the providers as tied and decide on the trial
- Test accuracy on 30 real questions and watch wrong answers on the hard 10; a chatbot's answers are your statements
How do you choose a chatbot provider? Score them against your needs, not their feature lists
Choose a chatbot provider by writing down what you need, weighting each need, scoring each shortlisted provider 1 to 5 against the same needs, and checking whether the winner survives a change in the weights. The reason to do it as arithmetic is that vendor comparison pages score every product on the vendor's own criteria. A weighted matrix puts your priorities in the formula, and shows you when two providers are effectively tied.
This guide gives you the criteria, a scored example for two very different buyers where the ranking flips, the data-handling questions you can ask based on what GDPR Article 28 requires of a processor, a way to test answer accuracy before signing, a cost model with a break-even point, and a two-week pilot plan. No providers are named: the process is what transfers, and any specific product comparison would be out of date in a quarter. For a list of tool types, see website chat tools; for what chatbots cost in total, see how much do chatbots cost.
Which criteria should you compare?
| Criterion | What to ask | Evidence to request |
|---|---|---|
| Answer control | Can the bot answer only from content I supply, and can I see which source it used? | A demo with your own documents; a list of questions it declined |
| Human handoff | When it cannot answer, does a person get the full conversation? | A test chat that escalates mid-conversation |
| Data handling | Where is data stored, how long, who are the sub-processors, can I delete it? | The data-processing agreement; sub-processor list; retention settings |
| Reporting | Can I see resolved, handed-off and unanswered questions? | A sample report on real data, not a marketing dashboard |
| Cost | What is the monthly price, and what changes it as volume grows? | A written price at your volume and at three times your volume |
| Setup effort | How many hours to the first live answer, and who maintains it? | A time estimate from someone who has done it, plus a named owner on your side |
| Integrations | Which systems does it read from or write to, and who builds that? | The list, with what is included and what is custom |
| Languages | Which languages does it answer in, and are they tested? | A test in each language you serve |
Not every row matters to every buyer. The next section shows how weights express that.
| Provider | Buyer A (small store) | Buyer B (regulated EU service) |
|---|---|---|
| X: template bot | 3.45 | 2.75 |
| Y: live chat plus AI | 3.75 | 3.45 |
| Z: custom build | 2.85 | 3.90 |
Weights for A favour cost and setup; weights for B favour data handling and answer control. Totals = Σ(weight × score) ÷ 100.
Worked example: a weighted matrix for two buyers
Method: give each criterion a weight so the weights add up to 100, score each provider 1 to 5 on each criterion, then compute Σ(weight × score) ÷ 100. A total of 5.0 would mean a perfect score everywhere.
Sample data: three anonymous providers, scored on six criteria after a trial.
| Criterion | Provider X: template bot | Provider Y: live chat plus AI | Provider Z: custom build |
|---|---|---|---|
| Cost | 5 | 4 | 2 |
| Setup effort | 5 | 4 | 1 |
| Human handoff | 2 | 4 | 3 |
| Answer control | 2 | 3 | 5 |
| Data handling | 3 | 3 | 5 |
| Reporting | 2 | 4 | 3 |
Two buyers with different needs:
- Buyer A: a small online store that wants something live this month on a low budget.
- Buyer B: a regulated service in the EU that must control what the bot says and where data goes.
| Criterion | Weight, buyer A | Weight, buyer B |
|---|---|---|
| Cost | 25 | 10 |
| Setup effort | 20 | 5 |
| Human handoff | 20 | 20 |
| Answer control | 15 | 25 |
| Data handling | 10 | 30 |
| Reporting | 10 | 10 |
| Total | 100 | 100 |
Computed totals:
| Provider | Buyer A | Buyer B |
|---|---|---|
| X: template bot | 3.45 | 2.75 |
| Y: live chat plus AI | 3.75 | 3.45 |
| Z: custom build | 2.85 | 3.90 |
How one total is computed: for buyer A, provider Y = (25×4 + 20×4 + 20×4 + 15×3 + 10×3 + 10×4) ÷ 100 = (100 + 80 + 80 + 45 + 30 + 40) ÷ 100 = 3.75. For buyer B, provider Z = (10×2 + 5×1 + 20×3 + 25×5 + 30×5 + 10×3) ÷ 100 = (20 + 5 + 60 + 125 + 150 + 30) ÷ 100 = 3.90.
The same three providers rank Y, X, Z for buyer A and Z, Y, X for buyer B. No provider is the best provider; one is best for a set of weights.
Sensitivity check. Move 10 points of buyer B's weight from data handling (30 to 20) to cost (10 to 20). Provider Z falls from 3.90 to 3.60 (it loses 10 × 5 on data and gains 10 × 2 on cost) and provider Y rises from 3.45 to 3.55 (loses 10 × 3, gains 10 × 4). The lead shrinks from 0.45 to 0.05. If a result flips after a 10-point shift, treat the two providers as tied and decide on the trial, not the matrix.
Weighted score, buyer B
out of 5.0What data-handling questions should you ask?
If the bot handles personal data about people in the EU, GDPR Article 28 sets out what must be in the contract between you (the controller) and the vendor (the processor). Turn that into questions:
- Instructions. Does the vendor process data only on your documented instructions?
- Sub-processors. Will they get your prior written authorisation, specific or general, before using another processor? With a general authorisation they must tell you about planned changes so you can object. Ask for the current list.
- Security and confidentiality. Are staff bound by confidentiality and are the Article 32 security measures in place?
- Data subject requests. Will they help you answer access and deletion requests?
- Deletion or return. At the end of the contract, will they delete or return all personal data, at your choice?
- Audits. Will they provide information to show compliance and allow audits, including inspections?
- Liability down the chain. The initial processor stays fully liable to you if a sub-processor fails.
A vendor that cannot answer these in writing has told you something. This is a checklist, not legal advice; ask your own counsel to review the agreement.
How do you test answer accuracy before you sign?
A chatbot's answers are your statements. In Moffatt v. Air Canada (2024 BCCRT 149), British Columbia's Civil Resolution Tribunal held an airline responsible for incorrect information its website chatbot gave a customer. Treat accuracy as a contractual risk, not only a quality metric.
Run this test with each shortlisted provider using your own content:
- Collect 30 real questions from your support inbox: 20 the content can answer, 5 it only partly answers, 5 it cannot answer at all.
- Mark each answer as correct, partly correct, wrong or declined.
- Calculate: correct answers ÷ 20 answerable questions; and wrong answers ÷ 10 unanswerable and partly answerable questions. The second number is the one that matters. A bot that confidently answers a question it should have declined is where liability comes from.
- Set a bar before you start: for example at least 17 of 20 correct (85%), and at most 1 wrong of the 10 harder questions. Adjust to your risk, but write it down first.
A provider with 18 of 20 correct and 4 of 10 wrong on the hard set is worse than one with 16 of 20 and 0 of 10 wrong, because the second declines when it does not know.
How does handoff to a human work, and how do you test it?
Ask for three things in the trial:
- Triggers. What sends a chat to a person: a visitor asking for one, low-confidence answers, or a keyword? Can you change it?
- Context. Does the agent see the whole conversation and the page the visitor was on, or does the visitor have to repeat themselves?
- Hours. What happens outside working hours: a message form, or a bot that keeps promising a reply?
Run the test yourself: start a chat, ask something outside the knowledge, and time how long it takes to get to a person with the transcript in hand.
What is the cost trade-off, and where is the break-even?
Providers charge in different shapes: a flat monthly fee, a fee per resolved conversation, or a mix. Compare at your volume.
Sample data: Provider P charges $400 a month plus $0.50 per resolved conversation. Provider Q charges $0 a month and $1.50 per resolved conversation.
| Resolved conversations per month | Provider P | Provider Q |
|---|---|---|
| 200 | $500 | $300 |
| 400 | $600 | $600 |
| 1,000 | $900 | $1,500 |
Break-even: 400 + 0.50x = 1.50x, so x = 400 resolved conversations a month. Below 400, Q is cheaper; above, P is. At 1,000 conversations P costs $0.90 each against $1.50 for Q. Add setup and upkeep hours (your time, valued at an hourly cost) to both before deciding.
What does a two-week pilot look like?
| Days | Task | Output |
|---|---|---|
| 1–2 | Load the same 20 documents into each shortlisted provider | Same inputs |
| 3–4 | Run the 30-question test | Accuracy and wrong-on-hard counts |
| 5–6 | Test handoff and out-of-hours behaviour | Time to human |
| 7–8 | Ask the data-handling questions in writing | Written answers |
| 9–10 | Score the matrix with weights agreed before the trial | Totals |
| 11–14 | Run the sensitivity check; run the winner live on a low-traffic page | Decision |
How TapCub's chat fits this checklist
TapCub's live chat is one kind of candidate: a human live chat that adds AI replies from a knowledge base, so it should be judged against the criteria above like any other.
- Answer control. In Live chat › Knowledge base, question-and-answer entries are sent as written without calling the AI; document entries are searched and used to draft a reply. Use the Playground to test without spending quota.
- Handoff. In Live chat › Chat settings › AI assistant, choose off, AI only when all agents are offline, or AI first (it stops when an agent takes over, replies, or the visitor asks for a person). With too little material, it hands over to a person.
- Reporting. Live chat › Reports shows conversations, resolution rate, first response and knowledge-base hit rate (the share answered by direct match or retrieval).
- Boundaries. AI replies only generate text. They do not change orders, send emails or call other systems, and the chat widget has no pre-chat form, so ask for an email in the conversation or use offline messages.
If you need a bot that takes actions such as refunds or order changes, TapCub does not do that; it measures chat outcomes and conversion once you have one in place.
FAQ
What should I look for in a chatbot provider?
Answer control, human handoff, data handling, reporting, cost at your volume, setup effort, integrations and languages. Weight them by your needs and score each provider on the same scale.
How do I compare chatbot providers fairly?
Use a weighted matrix: weights add to 100, scores 1 to 5, total = Σ(weight × score) ÷ 100. Then shift 10 points between two weights; if the winner changes, the providers are effectively tied.
What questions should I ask about data?
Whether data is processed only on your instructions, who the sub-processors are and how you are notified of changes, how data is deleted or returned at contract end, and whether audits are allowed. These mirror GDPR Article 28.
Is a company responsible for what its chatbot says?
In Moffatt v. Air Canada (2024 BCCRT 149) a tribunal held the airline responsible for incorrect information from its website chatbot. Test accuracy and wrong-answer rates before you go live.
How long should a chatbot trial be?
Two weeks is enough to load content, run a 30-question accuracy test, test handoff and score the matrix. Longer trials mostly delay the decision.
Sources
Was this article helpful?
We use your answers to fix and expand this page.
Couldn’t send. Please try again in a minute.
Thanks for the feedback. We’ll use it to improve this page.You’ve already sent feedback on this page. Thanks.

Product Experience Specialist
Writes about product experience, practical workflows, and growth, turning complex processes into clear steps for everyday decisions.
