Somewhere in your setup there is a model call doing a job that grep would do
better, and it runs on every commit. It was quicker to write that way, it worked
on the first try, and nobody has looked at it since.
The rule of thumb it breaks is worth stating on its own, because it gets ignored constantly:
Use a script as the first gate and an agent as the fallback.
What you are actually buying
This is the cut from a deterministic core with reasoning at the edges made one level down. Not which parts of the system get to reason, but which parts of a single check do.
The saving is real and worth having. It gets quoted because it is the easiest row to measure, and it is also the last one that should decide anything.
| an agent | a script | |
|---|---|---|
| the same question tomorrow | a plausible answer | the same answer |
| how often it is right | usually | every time, or it is a bug you can fix |
| why it decided that | a paragraph | a diff |
| what it costs | per invocation, forever | effectively nothing |
Read the rows in that order, because that is the order they matter in. A rule encoded as a script answers identically on the ten-thousandth run, and when it is wrong it is wrong the same way every time, which is what makes it fixable. An answer that varies cannot be debugged, only re-rolled.
The third row is what turns this from a preference into a requirement. When a script rejects your work you can read the logic, disagree, and change it in a pull request. When a model rejects it you get prose, and prose is not something a team can version, argue with precisely, or trust to say the same thing tomorrow. A gate whose reasoning cannot be inspected is a gate people learn to route around.
The tokens you stop burning are real, and they are a side effect. You did not convert reasoning into a script to save money. You did it so the answer stopped moving.
The order to try
- A script. If a rule can be expressed as a rule, express it.
- A script that escalates. Decide everything decidable, hand over only the residue.
- An agent. Only for the judgment no rule can express.
Most work stops at step one, and a surprising amount of what gets handed straight to step three belongs there.
The answer was always small
Here is the case that makes the argument concrete. Ask an assistant to improve a mutation score and it will do the obvious thing: open the report. A PIT run produces roughly two megabytes of HTML. Reading it costs something like forty thousand tokens, and what comes back is a summary that is usually vague and occasionally wrong, because summarising a large document is exactly the task a model does confidently and imperfectly.
The information you wanted was never two megabytes. It was a dozen lines:
$ .claude/tools/mutation_survivors.py --limit 10
PetValidator.java:41 negated conditional SURVIVED
PetValidator.java:52 removed call to reject SURVIVED
OwnerController.java:88 changed boundary SURVIVED
...
3 survivors in 2 classes, score 84% (threshold 80%)
Same finding, a few hundred tokens, and identical on every run. The report is generated by a build tool that already parsed the data into XML; the script reads the XML and prints the rows where a mutant lived. There is no judgment anywhere in that, which is precisely why no model should be doing it.
The general shape: the answer was always small, only the artifact was big. Wherever an assistant is reading a large generated file to extract a small fact, there is a script waiting to be written, and the script will be both cheaper and right every time rather than usually.
It also changes what you do next, which matters more than the tokens. When
deskspace first ran its mutation
gate it came back at 78%, and the report named every survivor in seven lines.
One was a method that could have returned true for every input and passed the
entire suite.
BookingRepository.java:39 isTaken BooleanTrueReturnValsMutator SURVIVED
Four test classes later it was 100%, and the number is not the point: without the script, “improve the mutation score” is a research task. With it, the work is a list.
Judgment does have a place in the same workflow, and it belongs next to the
script rather than instead of it. A document can say what a good test looks like.
Only the script can say that three mutants survived in PetValidator.
The same rule, one layer down
Hooks make this concrete, because most hook systems let you choose what runs.
{
"hooks": {
"PreToolUse": [{
"matcher": "Bash",
"hooks": [{ "type": "command", "command": "jq -r '.tool_input.command' | grep -qE '...'" }]
}]
}
}
That gate can be a command, a prompt, or an agent. A prompt hook asks a
model whether the command looks acceptable. A command hook runs jq and grep.
For a rule as crisp as “this flag is not allowed”, the model will usually agree
with you, and usually is the wrong standard for a gate. The grep is not
persuadable, does not have an off day, and answers the same on every tool call
for the life of the repository. Every gate in the repository this site is built
from is a command, and none of them needs to be anything else.
When the agent is the right answer
There is a real failure in the other direction: encoding a rule you have not settled, so the script fires on something reasonable people still disagree about. Judgment that resists encoding should stay judgment.
The useful part is that reaching for the agent is now a signal. It means one of two things, and both are worth noticing:
- the rule was never settled enough to encode, which is a conversation to have
- or you have just found the thing worth encoding next time, which is how the script grows
Treated that way, every call you cannot avoid is either a decision you owe someone, or tomorrow’s twenty lines of shell: one more answer that stops being a guess and starts being a fact.
If your team is paying an agent to do work a script did better, that’s the work I do.