Resolution, Not Deflection: The Chatbot Numbers Worth Watching
A chatbot that keeps people away from your inbox isn't the same as one that solves their problem. Five numbers to track, and how to count them by hand.
Published Updated 7 min read
The number that tells you whether a support chatbot works is its resolution rate: how many visitors got a correct answer and didn't need anything else. Deflection rate, the number most chatbot dashboards lead with, only tells you how many visitors didn't reach a human, and a visitor who gives up in frustration counts as "deflected" too. This post defines the five numbers I'd watch instead, and shows how to count each one by hand from your transcripts in about 20 minutes a week.
Why the industry is moving away from deflection
For years, support automation was sold on deflection: the share of conversations that never reached a person. It's easy to measure and it always looks good, because every conversation that ends without a handoff counts, including the ones where the visitor closed the tab annoyed.
That's changing. At a contact center summit this month, SiliconANGLE reported that success is being measured less by call speed and deflection and more by whether the customer's issue gets resolved. As analyst Zeus Kerravala put it there: "Those metrics don't … matter as much anymore."
Big contact centers have analytics teams to measure resolution. A small business usually has the owner and a spreadsheet. The good news: at small-business volume, you can measure it more honestly by reading than any dashboard can by guessing.
The five numbers
1. Resolution rate
What it is: the share of real conversations where the visitor's question was answered correctly and they didn't need anything more.
How to count it: take a sample of conversations from the past week (all of them if you have fewer than 30, otherwise 30 picked across different days). Mark each one with exactly one label:
- Resolved: the answer was correct and complete, and the visitor either said thanks, moved on, or asked a follow-up that was also answered.
- Handed off: the visitor asked for a person or left their details through the contact form.
- Unresolved: the answer was wrong, vague or missing, and the visitor didn't hand off.
- Not a real question: tests, greetings with nothing after them, spam.
Resolution rate = Resolved ÷ (all conversations − Not a real question).
The judgment call: a visitor who gets one answer and leaves without replying could be satisfied or could have given up. Read the answer. If it was correct and answered what they asked, count it as resolved. If you can't tell, count it as unresolved. Being strict with yourself is the whole point.
2. Wrong-answer rate
What it is: the share of answers that were factually wrong according to your own content and policies.
How to count it: in the same sample, flag every bot answer that states something untrue: a wrong price, a policy you don't have, a feature you don't offer.
Wrong-answer rate = conversations with at least one wrong answer ÷ real conversations.
This is the number to drive toward zero first. An unanswered question costs you a lead. A confidently wrong answer about a refund or a price can cost you money and trust. When you find one, fix the source: correct the page, or add a Q&A that states the right answer plainly. Don't try to patch it with clever bot instructions.
3. Handoff rate
What it is: the share of real conversations that ended with the visitor asking for a person or leaving their details.
How to count it: Handed off ÷ real conversations, from the same labels.
There's no single good value. Look at it next to the other numbers:
- High handoff and low resolution: the bot doesn't know enough. Add content.
- Low handoff and low resolution: visitors are giving up instead of asking for help. Make the handoff easier to find, and check that the bot offers it when it doesn't know.
- Low handoff and high resolution: what you want.
4. Handoff follow-through
What it is: of the visitors who handed off, how many got a human reply within the time you promised.
How to count it: for each lead in the period, note when it arrived and when you replied. Follow-through = replied on time ÷ total handoffs.
This one is about your team, not the bot, and it's where many chatbots quietly fail. A handoff that lands in an inbox nobody checks is worse than no chatbot at all, because the visitor did what you asked and still heard nothing. If you track only one thing about leads, track this.
5. The gap list
What it is: not a percentage, but a list: the distinct questions the bot couldn't answer this week.
How to count it: from your Unresolved and Handed off conversations, write down each question in the customer's own words. Merge duplicates. Count how many times each came up.
The gap list is your content to-do list. The questions at the top become Q&As, FAQ entries or new sections on your pages. A healthy chatbot's gap list gets shorter every week, and the questions on it get stranger.
What not to celebrate
Some numbers look like progress and aren't:
- Total conversations. More chats can mean more interest, or a confusing pricing page that sends everyone to the bot.
- Deflection rate on its own. As above, it counts frustrated exits as wins.
- Average conversation length. A long conversation can be a good, detailed answer or a visitor rephrasing the same question five times. Read it.
- Messages sent by the bot. That's your cost, not your outcome.
A 20-minute weekly routine
Here's the sheet I'd use. One row per conversation in your sample:
| Date | Page | Question (their words) | Label | Wrong answer? | Fix needed |
|---|---|---|---|---|---|
| Oct 20 | /pricing | "is there a monthly option" | Resolved | No | None |
| Oct 20 | /shipping | "do u ship to canada" | Unresolved | No | Add Q&A: shipping countries |
| Oct 21 | /services | "can I talk to someone about a bulk order" | Handed off | No | None |
Then, every week:
- Label 30 conversations (or all of them). About 10 minutes.
- Work out resolution rate, wrong-answer rate and handoff rate. One minute with a calculator.
- Check every lead from the week got a reply on time.
- Fix the top three items on the gap list and every wrong answer. About 10 minutes.
- Write the three percentages in a running list so you can see the trend.
Compare each week with your own previous weeks, not with benchmarks from vendor reports. Your traffic, your questions and your content are what matter, and the direction of the trend tells you more than any industry average.
When can you stop doing this every week?
Keep the weekly routine for at least the first month after launch, and restart it for a few weeks whenever something changes: new prices, a new product, a new policy, a redesign. Those are the moments a chatbot is most likely to quote something out of date.
Once the wrong-answer rate has been zero for several weeks in a row and the gap list is down to rare one-off questions, a monthly check is usually enough. The one thing I wouldn't relax is handoff follow-through. Check that every lead got a reply, every week, for as long as the chatbot is on your site.
Where ChatBento fits
In ChatBento's dashboard you can read every conversation and see every lead the chatbot collected, which is everything you need for the routine above. It doesn't calculate a resolution rate for you, and I'd be wary of any tool that claims to calculate it precisely: deciding whether an answer actually solved someone's problem still takes a person reading it, at least at small-business volume.
The short checklist
- Weekly sample of up to 30 conversations, labelled one way each
- Resolution rate, wrong-answer rate and handoff rate written down
- Every wrong answer fixed at the source
- Every lead replied to within the time you promise
- Top three gap-list questions turned into content
- Trend compared against your own past weeks
If you're still deciding whether a chatbot is worth it at all, the chatbot ROI calculator gives you a rough monthly figure for the support time it could save, and pricing starts with a free plan.