Anything below 80% needs addressing. That’s the chatbot resolution rate I’d hold any bot to. It only counts the conversations that ended with a positive outcome, where the problem was actually solved, and it leaves out every conversation the bot simply happened to be part of.
When someone tells me their AI agent is taking a lot of conversations, I want to know how many it solved. Taking them is the easy part. Someone who gives up halfway through and goes looking for a phone number has still been taken, and on a containment report they look exactly like a happy customer. Count the solutions and a lot of very good bot figures get a lot smaller.
That’s also the short version of my answer to whether AI will replace customer service. It’s a clear no, for two reasons. The first is the brand and the customers. Do you want your brand to feel like a machine, or something warmer? The second is technical. You can now make AI look and feel like a human, but so far it still can’t give good service. I don’t mean it can’t provide solutions. It can, and on the right questions it’s quick. What it often lacks is the service around the solution.
Support is not all about delivering a solution fast. People need to feel they matter, that you care and want to look after them. AI isn’t going to solve all your support problems, and the people predicting it’ll replace support teams are counting the answers a bot gives without asking how the person on the other end felt. These are the questions I’d ask before believing them.
Ask for the chatbot resolution rate, not the containment number
Put this to whoever runs the bot, whether that’s your own team, an outsourced support provider or a support partner. If all you get back is a containment figure, that’s a bad sign. The bot handled this many conversations, this share never reached an agent. Nobody asked how many of those people were helped, so nobody knows.
What you want to hear about is solutions. How often is the bot delivering a positive outcome, and how did they decide a conversation counted as one? If they can give you the chatbot resolution rate and explain how it’s measured without reaching for the containment slide, you’re talking to someone who’s looked properly. I’ve written before about why containing a conversation doesn’t mean resolving it, and the arithmetic is the same on an ordinary Tuesday as it is in peak season.
Below 80%, something has to change. Say your bot sits at 60. I wouldn’t switch it off straight away, and I wouldn’t leave it running and hope for the best either. I’d look at the questions it’s getting and why it can’t help with them. Sometimes it needs more training on things it should be able to answer. Sometimes it’s being asked things it shouldn’t be handling at all, and the fix is to point people to a human sooner. Either is fine. Leaving four in ten customers to work out for themselves that the bot can’t help them is not.
The same number decides how far the bot goes. I’d never go all in and push it. Start with the most common simple questions, the ones people would be happy to talk to an AI about and that it can solve easily and fast. Start with a small number of them and get those working well, so people learn to trust it. Then move up the issues one step at a time, making sure the solution rate stays over 90% before anything new goes in. So 80 is where I’d start fixing things, and 90 is where I’d want it to sit before it gets more to do.
Track the chatbot resolution rate for each type of question separately, not only as one blended figure. If the rate drops when you add something new, that question goes back to people until you’ve worked out why.
Find the requests simple enough to hand over
Before anything is automated, go into the ticket data and look at what’s being handled fast and often. Those are your candidates. A password reset. “Where is my package.” Requests where the need is clear and simple, and the customer doesn’t need looking after so much as a link or a tracking number.
The point of moving those to AI is that users get the response they need without waiting for a person, if that’s what they want. And it has to be what they want. Someone who’d rather ask a human about their parcel should still be able to, and the way to a person should be easy to find. Bots that trap people undo whatever the bot was meant to fix.
Watch out for a list built from what the business would like to be rid of, rather than what the data says is quick and frequent. If a request regularly needs a long back-and-forth with an agent, it isn’t simple, however simple it sounds in the meeting.
Work out what the bot actually saves
A finance director or a board will ask this one, and it’s easy to answer wrongly. The bot only saves money on the questions it solves. Measure the chatbot resolution rate, not the number of questions it receives.
Say a bot takes a thousand conversations in a month and solves six hundred. The savings case is built on six hundred. The other four hundred still reach a person, usually later and a bit more annoyed, having already explained the problem once to something that couldn’t help. A cost case that counts all thousand is counting money that was never saved.
Klarna is the public example I’d point a board at. They went hard on AI in customer service and have since reversed course, saying customers like talking to people. I’d rather a business found that out from the numbers on a small rollout than from its customers.
Check whether customers want a machine for this one
Ask it about every request type on the list, and ask it the way a customer would. Where real service is needed, someone upset or someone whose plans depend on the answer, your brand needs to be there, and that means a person. A bot that gives the right answer coldly has solved the problem and still left the customer feeling they don’t matter.
Hire for what’s left once the easy ones have gone
Once the bot is taking the password resets and the parcel questions, what reaches your people changes. Less of the work follows defined scripts and simple known problems, and more of it is complex, needing a deeper understanding of the product and the customer. So you need more skilled people, but fewer of them.
That changes who you hire. Someone who was good at getting through a high volume of short scripted contacts won’t necessarily be the right person for a full day of hard ones. I’d look for people with the attitude to dig in, the ones who say “I’ll find a way” instead of “I don’t know”, and then give them the time and support the job now needs. Stop judging them on handle time too. The contacts that are left are exactly the ones that shouldn’t be rushed.
Humans and AI will work side by side. AI will take more and more as it improves and gets better connected to the systems behind it, and I’d expect the 80% line to be easier to clear on a wider range of questions than it is today. The complex areas and the more specialised brands will still want the human touch, and I think premium brands will put a human first, because their customers will expect to feel looked after. That’s why my answer is still no.
For now, pull last month’s bot conversations and count the ones that ended in a solution. That’s your chatbot resolution rate. If it’s under 80, start with the questions it’s failing on.
About the Author
I started turning spanners at sixteen and went on to lead global support for some of the world’s most demanding industries. I’ve built a reputation for challenging the traditional ways of doing things, and pushing the limits of what’s possible.
From transforming operations at Volvo to building human-centred support models at Sigma Technology and now FIXATE, I lead with clarity, care, and courage. I see the big picture, sweat the details, and always try to do what’s right, even when it’s hard.
If something’s worth doing, I want to do it properly. And I’m probably already thinking about how to make it better.
Read more from the author
