You ask for the same change twice and get two different implementations. Not one right and one wrong: two that both compile, both pass, and differ in structure. Somewhere around the third time this happens, most engineers file the tool under flaky and stop trusting it.
That reaction is reasonable and it is aimed at the wrong thing. Every tool you have trusted for twenty years is a function. Same source, same flags, same binary, and when that stops being true you have found a bug worth reporting.
A language model is not a function in that sense. At each step it produces a probability distribution over the next token and something samples from it. Two runs walk two paths through the same distribution. The spread is not a defect that better engineering will remove: it is the mechanism.
What actually moves the spread
Three settings do most of the work, and they are worth knowing by name because the vocabulary keeps coming back.
| lever | what it does | what it costs |
|---|---|---|
temperature |
flattens or sharpens the distribution before sampling | at 0 you get the highest-probability path, which is repetitive and often worse at anything requiring a leap |
top_p |
samples only from the smallest set of tokens covering probability p
|
narrower than temperature alone, same tradeoff |
| a fixed seed | pins the sampler, where the provider exposes one | not offered everywhere, and rarely survives a model version change |
Setting temperature to zero is the move everyone reaches for first. It narrows
the spread and it does not close it: batching, hardware and floating point
non-associativity mean identical inputs can still take different paths. Treat
temperature: 0 as narrower, never as reproducible.
Measure your own spread before you argue about it
Opinions about how variable these systems are get very confident and very unfounded. The number is cheap to obtain for your own setup, and it is the only version of the number that means anything.
#!/usr/bin/env bash
# spread.sh: run one prompt N times, report how many distinct outputs came back.
# Point RUN at whatever you actually use. It must read a prompt on stdin and
# write the result to stdout.
set -euo pipefail
PROMPT_FILE="${1:?usage: spread.sh <prompt-file> [runs]}"
RUNS="${2:-10}"
OUT=$(mktemp -d)
for i in $(seq 1 "$RUNS"); do
"${RUN:?set RUN to your model command}" < "$PROMPT_FILE" > "$OUT/$i.txt"
done
echo "distinct outputs: $(sha256sum "$OUT"/*.txt | awk '{print $1}' | sort -u | wc -l) / $RUNS"
echo "line-level spread:"
diff "$OUT/1.txt" "$OUT/2.txt" || true
Run it against a prompt you actually use. Then run it again with the task specified twice as tightly, and compare. The second number is the one that matters, because it is the one you can change.
The move that actually works
Chasing reproducibility at the model is fighting the mechanism. The productive move is to stop needing it.
A deterministic pipeline gives you one output and you check it once. A sampling one gives you a spread, so you specify the boundary that output has to land inside and you make crossing that boundary mechanical. Tests, types, a linter, a schema, an architecture rule, a CI gate: these are all the same move, which is converting “looks right” into “passes or does not”.
# The question is not "did it produce the same code", it is
# "did it produce code that lands inside the boundary".
gates:
- build passes
- tests pass, including the ones written before the change
- public API unchanged unless the task said otherwise
- no new dependency without an explicit line in the task
- architecture rules still hold
Every one of those is checkable by a machine, and none of them cares which of the five plausible implementations came back. That is the point. You are not trying to get the same answer twice. You are trying to make any acceptable answer obviously acceptable and any unacceptable one obviously not.
The trade underneath
Everything above is one instance of a decision you make dozens of times a week: for this task, do I write a program or do I invoke reasoning?
A parser over a well-formed log is free, instant and identical on the ten-thousandth run. A model reading the same log costs tokens every time and gives you a slightly different summary. If the format is stable, the parser wins and it is not close. If the format is whatever six teams felt like emitting, the parser is the thing you rewrite every month and reasoning wins.
| a program | reasoning | |
|---|---|---|
| stable, structured input | identical, free, instant | tokens per run, spread per run |
| input nobody agreed a format for | rewritten monthly | absorbs the change |
| the rule is known | encode it | wasteful |
| the rule is what you are trying to find | premature | this is the job |
The failure runs in both directions. Piping a well-formed markdown file through a model on every invocation to pull out three fields is paying rent on a solved problem. Hand-rolling five hundred lines of parser for something genuinely ambiguous is the same mistake facing the other way.
What makes this compound is the middle path:
Reason once, encode, replay. Use reasoning to derive the rule. Then write the rule down as a script, a schema, a command, a gate. Every reasoning act should ideally leave behind a deterministic artifact you never have to reason about again.
That is the whole series in one line. The parts that follow are the same trade made somewhere else: in a prompt, in a script, in a command, in a hook that refuses before the act rather than complaining after it.
If your team is still arguing about whether the tool is reliable instead of about where the gates go, that’s the work I do.