Every AI coding assistant ships a dashboard, and every dashboard leads with the same number: acceptance rate. The percentage of suggestions your engineers accepted.
It is a wonderful metric, for the vendor. It goes up when the tool is used more, it is easy to instrument, and it is almost entirely disconnected from whether your organisation ships anything faster.
I keep meeting engineering leaders who have a year of this data and still cannot answer the only question their CFO is asking: did it work?
Why acceptance rate misleads
An accepted suggestion is a suggestion that looked plausible enough to keep in the moment. That is all it means.
It does not mean the code was correct. It does not mean it survived review. It does not mean it was still in the codebase a month later. And it definitely does not mean the engineer was faster than they would have been typing it themselves, and plenty of accepted suggestions get accepted, then rewritten twice.
The metric also moves for reasons that have nothing to do with value. A team working in a well-trodden framework will see high acceptance because the model has seen a million examples of that code. A team working on your genuinely novel domain logic, the code that actually differentiates your business, will see low acceptance.
Read naively, the dashboard tells you your most valuable team is your worst performer.
The measurement that actually matters
The useful question is not “how much AI-generated code do we have?” It is “has our ability to deliver changed?” Which means measuring the system, not the tool.
Start with a baseline before you change anything. This is the step almost everyone skips, and its absence is why so many rollouts end in an argument rather than a conclusion. You cannot demonstrate improvement against a number you never recorded.
The four delivery metrics are the right starting frame, because they were designed to resist exactly this kind of gaming:
- Lead time for change: commit to production. If assistants are helping, this is where it shows up. If it hasn’t moved in six months, the leverage is going somewhere other than delivery.
- Deployment frequency: are you shipping more often, or just producing more code that queues up behind the same release process?
- Change failure rate: the counterweight. Faster output that breaks more often is not a win, and this is the metric that catches it.
- Time to restore: whether the team still understands the system well enough to fix it under pressure.
To those I add two that are specific to assisted development:
- Review latency and review depth. Output volume rises immediately; review capacity does not. Watch the time-to-first-review and the number of comments per hundred lines. When comment density falls while volume rises, people have quietly started approving code they have not read. That is the single most expensive failure mode in this whole space, and it is invisible on a vendor dashboard.
- Code survival. What fraction of merged code is still there in ninety days? Churn is not automatically bad, but a sharp rise in code that gets written, merged, and then rewritten is a strong signal that the first draft was plausible rather than correct.
Expect the dip
Here is the thing nobody selling you a licence will mention: teams usually get slower first.
There is a real learning curve in knowing which tasks are worth delegating, how much context to supply, and, hardest of all, how to recognise confident nonsense quickly. Engineers who have not built that judgment yet will spend more time reviewing bad output than they would have spent writing good code.
If you measure at week three and conclude it failed, you cancelled during the dip. If you measure at week three and conclude it succeeded because acceptance rate is 40%, you have learned nothing at all. Give it a quarter, and measure the things above.
The uncomfortable part
Sometimes the honest answer is that it did not help much.
I have seen teams where assistants produced a genuine step change, and teams where twelve months of licences produced a rounding error. The difference was rarely the tool. It was whether the codebase had clear enough boundaries for an assistant to work in safely, and whether the team changed its practices to match, or just turned it on and hoped.
An assistant amplifies whatever is already there. Point one at a well-structured system with a strong review culture and you get leverage. Point one at a tangle of implicit coupling and undocumented conventions and you get the same tangle, faster.
That is not a reason to avoid the tools. It is a reason to be honest about what you are measuring, and to fix the substrate before blaming the model.
Fixing the substrate is the unglamorous half, and it starts where writing a program instead of asking for reasoning starts: with the conventions your team keeps re-explaining, and what it takes to have a machine check them instead of a person remembering to.
If your organisation is trying to answer this question and the data isn’t cooperating, that’s the work I do.