You approve a small automated edit to one of your own files. It reports success. The file it leaves behind does not parse, nothing errored on the way through, and the two lines that did the damage look correct:
assert anchor in s
s = s.replace(anchor, new, 1)
Read the assertion again. It tests existence. The invariant the code actually
depends on is uniqueness, and those come apart in exactly one case: when the
anchor appears more than once. So the assertion passes precisely when it should
have stopped you, and replace with a count of 1 cheerfully patches whichever
occurrence it met first, which was in a different branch of the file.
The tool had no way to object
str.replace cannot refuse. It is not defective, it is general: given an
ambiguous instruction it picks one and returns. There is no input for which it
raises, no flag that makes it strict, no way to say “only if this is
unambiguous”. Ambiguity is not an error condition it knows about.
An editor that requires a unique match refuses the same instruction outright. Same job, one less way to be silently wrong.
| the job | permissive | refuses |
|---|---|---|
| replace text | s.replace(a, b, 1) |
an edit that requires a unique match |
| read a config key | d.get("timeout") |
a schema that fails on an unknown or missing key |
| use a variable | unset expands to empty | set -u |
| parse a date | a format guesser | an explicit format that raises |
The pattern is the same every time: the permissive option is more powerful, and what you pay for that power is an invariant it no longer enforces on your behalf. Reaching for a script before an agent is that trade made about which tool does the work. This is the same trade made about which tool is allowed to accept the input.
Supply the invariant the tool lacks
When the general tool is genuinely the right one, the invariant does not disappear. It just becomes yours to state:
assert s.count(anchor) == 1, f"anchor occurs {s.count(anchor)} times"
That is one word different from the version that failed, and it is the whole
difference. in answers “is it there”. count == 1 answers the question the
code was actually built on.
Better still, put it somewhere it cannot be forgotten next time:
$ python3 scripts/anchored_edit.py check_post.py \
--anchor-file a.txt --replacement-file b.txt
refusing to edit check_post.py: anchor occurs 2 time(s), expected 1.
An ambiguous anchor is how the wrong branch gets patched. Lengthen it
until it is unique, or pass --expect if you really mean all of them.
Twenty lines, and the class of mistake stops being available.
The one that proves it
The same week, a different check: whether any line in a diagram runs through a text label. That had been worked out by hand four times, once per diagram, in a browser console. It caught something every time, which felt like evidence the manual approach was working.
Encoding it as a script took one pass. On its first run it found a stroke through a label in a diagram that had already been published, because that was the one diagram nobody had checked by hand. Then, when the first fix moved the label into a second line, it caught that too, immediately, with no browser round trip.
Four careful manual passes missed what one script found in a second. That is not a story about carelessness. Manual checking is sampling: it covers what you remember to look at.
Two corollaries worth stealing
Assert the invariant, not a proxy for it. An assertion that passes in the dangerous case is worse than none, because it buys confidence without buying safety.
Revert, do not repair. When an automated edit corrupts a file, git checkout
and redo it. Repairing a file you can no longer reason about is how one broken
edit becomes three, which is the shape of the second failure above.
If your team keeps finding the same class of bug and fixing it one instance at a time, that’s the work I do.