My dashboard said 95%. The honest number was 89%.

On Friday morning my own dashboard told me our AI's automated resolution rate was 95%.

I built that dashboard. I wanted to post the number. Instead I checked it — because a week earlier I'd learned the hard way that a support-AI percentage without its definition attached is a trap you set for yourself.

The honest number was 89%. And the more useful sentence was longer: 89% of the conversations our AI answered were resolved with no human takeover — but it only answered a small share of everything that came in.

This post is about the gap between those two sentences, because the whole AI-support industry lives in it.

The bug: deleting your failures from the denominator

Our first version counted a conversation toward "automated resolution" if its last reply came from the AI and the customer went quiet. Sounds reasonable. It isn't.

Think about what happens when the AI answers, the customer pushes back, and a human agent has to step in. Under the last-reply rule, that conversation's final message is now human — so it silently drops out of the metric entirely. It doesn't count as an AI failure. It just vanishes.

Same numerator, smaller denominator. On our data, that one detail inflated the rate from 89% to 95%. On email alone, from 71% to 85%. Nothing was faked; the definition just quietly excused every failure.

The fixed rule is simple: if the AI answered, the conversation stays in the math forever. A human taking over afterwards counts against the rate. A customer writing back later — even weeks later — voids the resolution.

I'd call this an embarrassing bug, except for what I found when I went to check how the big vendors count.

What "automated resolution" means at the vendors

I read the primary documentation — help centers, pricing terms, the pages that define what customers get billed for. The definitions are all different, and some of them are doing on purpose what my bug did by accident.

Intercom Fin counts a resolution when the customer confirms the answer helped, or simply disengages for 24 hours after Fin's last answer — an "assumed resolution." A teammate replying afterwards does not void it; only the customer explicitly asking for a human does. Reopens get deducted. And as of July 2026, conversations where Fin was active but couldn't answer are excluded from the denominator too — a change that raises the headline rate without a single customer outcome improving. One Fin customer publicly audited 40 conversations marked "resolved by Fin" and found 7 actually resolved: 17.5% real against 40% billed.

Zendesk is stricter: 72 hours of silence, an LLM double-checks that the request was actually satisfied, and any escalation to a human disqualifies the conversation — they bill per resolution, so the definition is contractual.

Ada is strictest: any human involvement disqualifies, and their own AI grades the conversation as relevant, accurate, and safe before it counts.

So when you see "67% resolution rate" and "70% automated resolution" from two different vendors, you are not looking at the same measurement. You might not even be looking at the same kind of measurement.

The number nobody puts next to the number

Every one of those rates shares a quieter assumption: the denominator is conversations the AI engaged, not everything your customers sent. That's fair — a rate needs a base — but it means the headline is meaningless without a second number: involvement.

A 90% resolution rate at 5% involvement resolves 4.5% of your inbound. A 55% rate at 95% involvement resolves ten times more. The first one produces a much better slide.

That's why our dashboard now refuses to show the rate alone. The tile reads the whole sentence: the rate, and the share of inbound the AI answered. The unflattering half of the sentence is the strategy: our resolution rate is already strong; the work is safely raising how much the AI is allowed to answer. Which is exactly what our thresholds, canary tests, and improvement engine spend all day on.

The rules we now hold every metric to

Three rules, enforced in code, after getting burned:

1. The denominator is decided before the outcome is known. Any metric that removes items based on how they ended is survivorship bias wearing a percentage sign.
2. Every number carries its own definition, window, and sample size. A tile that says "42%" without saying of what, over when, out of how many is decoration.
3. A rate never appears without its base rate. Resolution comes with involvement. Acceptance comes with volume.

And because definitions drift, there's a statistical tripwire behind it all — a change detector that accumulates evidence per channel and flags degradation before a human would spot it in the noise, with automatic rollback for the changes our improvement engine ships itself.

Questions to ask any vendor quoting a resolution rate

If you're evaluating AI support tools, these five questions will tell you more than any demo:

1. What counts as "resolved" — customer confirmation, or silence? How long is the silence window?
2. If a human teammate replies after the AI, does the conversation still count as AI-resolved?
3. What's the denominator — conversations the AI engaged, or all inbound? Has that denominator's definition changed recently?
4. What happens when a "resolved" customer comes back a week later?
5. Can I see the involvement rate next to the resolution rate?

Any vendor with honest math will answer in one breath. Ours is on the dashboard, next to the number, with the sample sizes attached.

The 95% would have looked great on a slide. The 89% is the one I can defend — and the one that actually tells me what to build next.