The release that found out the central claim was false, and fixed it.
±15% had been printed on every report for eight releases, with every dollar
figure descending from it, and nothing in this repository established that it
held. The first run of measure-token-band.mjs against the official counting
endpoint found it did not: the numeric sample was 30.6% under, Spanish prose
22.1% under. Nine of eleven samples underestimated, always in the direction that
under-reports cost.
Two things were wrong. Digits were counted at three per token where Claude splits them far more finely — corrected in isolation, that sample went to -5.0%. And the estimator turned out to be calibrated for English specifically, not for prose: German measured -37.3% under one divisor that served every language.
The band is measured now, at ±15%, and it landing back on the old number is a coincidence rather than a restoration: that 15 bounded nothing, and this one bounds twenty-one samples across seven languages and six text types, worst case 11.2%. Every language divisor has a held-out test in a different register.
Also here: trazum baseline and a check that gates on drift rather than only on
a ceiling; a pull-request comment that leads with what the branch costs; and
below-cache-minimum no longer asserting from an estimate near a hard threshold,
which was wrong advice rather than an imprecise figure.
The sections below are as they accumulated, entry by entry, and were not consolidated: they are the record of what happened in the order it happened.
Fixed#
SECURITY.md claimed prompts are never stored, and that stopped being true
when --suggest --cache shipped. The sentence was written when it was
unqualified and correct; the cache arrived later and nobody went back to the
security document. It writes the model's raw response, keyed by the prompt, to a
file under the user's home directory.
Nothing about the feature is wrong — it is opt-in, it is local, and the files
are 0600 in a 0700 directory precisely because a prompt is the most sensitive
thing this tool touches. What was wrong is a security document telling a reader
that no such file exists, which is the kind of error that survives review because
it reads as reassurance.
The paragraph now says what is guaranteed (nothing about a prompt reaches a server that keeps it) separately from what the CLI can be asked to do on the machine it already runs on, and names where to delete it.
The repository-hardening checklist listed a step that is done and enforced.
Pinning every third-party action to a commit SHA was item 6, phrased as
something to run once there is network access. It has been done for several
releases and security.test.js fails any uses: naming a tag or a branch — so
it is not a checklist item, it is an invariant, and it moved to a section that
says so. The list also had two items numbered 3 and a closing line counting
five of six.
Changed#
The corpus went from eleven samples to twenty-one, and every divisor now has a held-out test. The eleven that set the band left three languages calibrated on a single sample each, which is a fit rather than a measurement: a divisor chosen to minimise the error on one file will always look good on that file.
Ten samples were added, in two rounds. The first round gave Italian, Portuguese and Dutch a sample each — three languages that had a divisor by inheritance and no evidence — and the second gave every calibrated language a second sample in a different register: the first set are support prompts, the second are code-review prompts, different vocabulary and different length. That second sample is what turns the first from a fit into a finding, and all seven held:
calibrated on held out on
english +1.0% +0.4%
german -9.2% -8.5%
french -1.2% -5.8%
spanish -6.2% -9.7%The divisors moved as a result — Italian and Dutch had been taking English's 4 —
and the corpus-wide worst case is unchanged at 11.2%, on Japanese, which no
divisor touches. Nothing in twenty-one samples is outside ±15%.
Italian had to be rebuilt, and the reason is worth recording. Its first
function-word list was half Spanish: per con del una sempre are as common in one
as the other, so they earned nothing, the margin rule tied, and an Italian
code-review prompt came back null, fell through to the English divisor and
measured -21.9%. The detector was working exactly as designed — it refuses rather
than guesses — and the fault was a word list that could not tell the two apart.
The replacement is words Italian has and Spanish does not. Over-correcting it
broke the prose sample instead, which is the shape of this whole file: a list
tuned on one register is a list tuned on one register.
What a hundred samples per language would have bought, and why they were not written. The counting endpoint is free, so the constraint was never money. It is that every sample here was written by the same hand, and ninety more of those is ninety more of the same bias — a tighter-looking number resting on nothing new. Twenty-one real prompts bound the band honestly; a hundred invented ones would bound it decoratively. The corpus grows one sample at a time now, and the samples worth adding are the ones that came from somebody's actual work.
Fixed#
below-cache-minimum was asserting from an estimate, and near the threshold
that made it wrong advice. It compares the stable prefix against a hard limit —
512 tokens on Claude Opus 5 — and then tells the reader caching will not work
here. The prefix is estimated, so a prompt measured at 505 tokens could really be
at 540 and cache perfectly well. Not an imprecise figure: a reader told to stop
looking at the single largest saving Trazum offers.
Near the line it now says so, and names the way to settle it — --exact-tokens,
against an endpoint that is free. Far below the line it stays quiet, because a
hedge on every case is a hedge nobody reads.
And it does not hedge a number the caller measured. count defaults to
estimateTokens; a caller who supplied their own counter has an authoritative
figure, and telling them it might be wrong is its own kind of dishonesty — it
pushes them toward a check they have already done.
Both directions mutation-tested: hedging always fails the exact-counter test, hedging never fails the near-threshold ones.
This was listed as known, not fixed one release ago. The window shrank when language detection took the worst estimate from −37.3% to −9.2%; it did not close, which is why this is a fix rather than a note.
Fixed#
The estimator was calibrated for English and silently wrong for every other Latin language. Measured across eleven samples: German −37.3%, Spanish −22.9% and −22.1%, French −15.1%, English +1.0%. Nine of eleven underestimated, always in the direction that under-reports cost. Characters per token says why — English 3.44, French 2.66, Spanish 2.53, German 2.02 — while one divisor of 4 served all of them.
estimateTokens now detects the language and divides accordingly:
german-prose -37.3% → -9.2%
spanish-unaccented -22.9% → -6.5%
spanish-prose -22.1% → -6.2%
french-prose -15.1% → -1.2%
worst in corpus 37.3% → 11.2% (Japanese, untouched)The published band drops to ±15% — the measured worst case rounded up, the same rule that briefly made it 25. Landing back on the number that was a guess for eight releases is a coincidence, not a restoration: that 15 bounded nothing, and this one bounds eleven samples across four languages and six text types.
The signal is not accents, and that was tested rather than assumed. A Spanish
sample with zero diacritics measured −22.9% against −22.1% for accented Spanish,
which killed the hypothesis the previous release recorded. What separates these
languages is which words they are made of, so language.ts counts function words
— the of and to against der die und ist against que los las del.
It answers null when unsure, and most of its tests are about earning that.
A three-line prompt, a JSON schema, English instructions wrapped around a Spanish
example: no answer is safe for any of them, and a wrong language applies another
language's divisor to text that does not want it. null falls back to the English
divisor, which is what the estimator always did. Two bars guard it — four distinct
function words minimum, and a 1.6× margin over the runner-up — and removing either
one fails the suite.
One caveat stated plainly rather than buried: the four Latin divisors are calibrated on one or two samples each, so their residuals are in-sample and optimistic by construction. The band is set by the seven samples nothing was fitted to. The honest test is the next held-out sample in Spanish, French or German, and the corpus grows one sample at a time now.
Known, not fixed#
below-cache-minimum compares an estimated prefix against a hard 512-token
threshold, so an underestimate can report "caching will not work here" when it
would — wrong advice, not just an imprecise figure, and it costs the reader the
largest saving Trazum offers. The worst estimate on measured text is now −9.2%
rather than −37.3%, which shrinks the window considerably but does not close it.
The advisory should hedge near the threshold; that is its own change.
Added#
The corpus can grow now, and three samples were added to falsify a hypothesis. Measuring the band left one finding unexplained: Spanish prose comes out 22.1% under while English comes out 1.0% over, and accents are not the cause — weighting them from 2 to 5 moves the figure three points. The candidate explanation is merge-table coverage: text that is not English costs more tokens per character whatever its diacritics.
spanish-unaccented.txt is the test that can prove that wrong. It is Spanish with
zero accented characters, so if accent density were a usable detector for "this
is not English" — and on the old corpus it separated perfectly, 0.00% against
1.71% — this sample would slip past it and stay underestimated. If instead it
lands where accented Spanish lands, the phenomenon is the language and the accents
were a coincidence of one file. french-prose.txt (1.83% accented) and
german-prose.txt (1.64%) say whether other Latin languages behave like Spanish
or like English.
Adding them required fixing something first. The freshness digest covered the whole corpus, so it could not tell an edited file from an added one and answered both with "re-run the script" — correct for an edit, wrong for an addition, because it retires eight measurements that cost an API call each to admit one new sample. The corpus was effectively frozen: growing it was gated on a key nobody wanted to spend.
Digests are per sample now, via digestOfOne, and the two cases get what each
deserves. A file that changed since it was measured fails — its measurement
describes different text, which is the dangerous case because it passes while
being wrong. A file with no measurement skips out loud and is named, with the
command to run, because a gap in coverage is something to report rather than a
reason to distrust what has been measured. The existing fixture was migrated
without new API calls, which is sound only because the whole-corpus digest still
matched: that match is the proof those eight files are the ones that were
measured.
One guard had to be loosened to its intent rather than its letter: it asserted
import { digestOf } literally and failed the moment digestOfOne was imported
alongside it — a guard that breaks when you use more of the thing it protects.
A global correction factor was tried and rejected. The best available (×1.05) takes the worst case from 22.1% to 18.2%, and does it by pushing Japanese from +11% to +17% and English from +1% to +6%. That is redistributing error, not reducing it, and it damages the two samples the estimator gets right.
Fixed#
The ±15% error band was never true, and now it is measured. Eight releases
printed it on every report and every dollar figure descended from it, while
nothing in this repository established that it held. The first run of
scripts/measure-token-band.mjs against the official counting endpoint found two
of eight samples outside it, both underestimating:
numeric-heavy estimated 277, actual 399 -30.6%
spanish-prose estimated 352, actual 452 -22.1%Underestimating tokens means under-reporting cost. Trazum was telling people their prompts were cheaper than they are — the flattering direction, and the worst one for a tool whose whole argument is honest cost accounting.
One constant was simply wrong. Digits were counted at three per token; Claude splits long runs far more finely, because a merge table cannot cover every number. Correcting it in isolation — nothing else touched — takes the numeric sample from −30.6% to −5.0% and moves no other sample more than four points.
The Spanish error is not about accents, and that matters. Weighting accented characters from 2 to 5 moves that sample by three points. Spanish words tokenize into more tokens than English words of the same length even when they are pure ASCII, because Spanish is thinner in the merge table — so no per-character-class constant can fix it, and one Spanish sample cannot calibrate a signal that would. English prose lands at +1.0%, which says the estimator's structure is sound and its coverage of non-English text is not.
So the published band is now ±25%: the measured worst case (22.1%) rounded up,
because eight samples cannot bound a worst case tightly and the honest direction
to be wrong in is the pessimistic one. It is ESTIMATE_ERROR_BAND_PCT, exported
from the core — it had been a literal in twenty-three files with its only
machine-readable copy in a test, which is why correcting it meant a hand sweep
across three locale catalogues, four READMEs, the MCP tool descriptions, the web
app and the demo. A guard now fails when any file states a band the code does not
publish, with CHANGELOG.md excluded because rewriting history to match the
present is the opposite of a changelog.
The roadmap predicted CJK would be the problem. CJK is fine at −3.2% and +11.2%. It was wrong about which text type and right that one number cannot cover all of them.
Changed#
1.8.0 is on npm. @trazum/core, @trazum/cli and @trazum/mcp were
published by hand on 2026-08-13. The first publish had to be manual — a trusted
publisher is configured on a package's settings page, and that page does not
exist until the package does — so every release after this one goes through a tag
with no credential anywhere.
The documentation caught up: the "not published yet" notes are gone from
RELEASES.md, the front page and both package READMEs, and Getting started
leads with npx @trazum/cli instead of instructions for building from source,
which are now folded away for people working on Trazum itself.
The guard that watched those notes was keying on the wrong thing, and the
publish proved it. It asked whether v1.8.0 was tagged, on the reasoning that
release.yml publishes on a tag and nothing else — checkable offline, which a
test in CI should be. But the first publish could never go through a tag, so no
tag was pushed, and the repository went on telling every visitor that nothing was
installable while three packages sat on the registry. That is the second signal
this claim has outlived; the first asked whether the changelog had a heading for
the manifest version, which is a release cut here rather than a package on npm.
There is no third proxy. Publication does not reverse, so the assertion is
one-directional now: no file may claim nothing is published. docs/releasing.md
records both traps the real publish hit — a 404 Not Found - PUT is npm hiding
an auth failure behind a missing-scope error, and npm view 404s for minutes
after a successful publish because the packument propagates behind the tarball.
Fixed#
npm was silently rewriting the manifest of both published binaries. Every
npm publish answered "bin[trazum]" script name was cleaned — npm stripping
the ./ from "./dist/index.js" and uploading a manifest that differs from the
one in this repository. On npm 12 the same correction reads "was invalid and
removed", which would put a package declaring a bin and carrying no executable
on the registry: npx @trazum/cli would resolve and then do nothing.
Caught on the first real publish attempt, in the wall of npm notice lines
nobody reads during the one command this repository cannot take back. Both
manifests now say dist/index.js, which npm has nothing to correct, and a guard
asserts it for every publishable workspace — plus that the target actually
travels in the tarball, since a bin pointing outside files is the same defect
arriving by a different route.
Fixed#
The CLI test suite passed in CI and failed on a contributor's laptop. Seven
tests, the first time anybody ran npm run verify on a machine whose locale is
not English — which was during a release, on the maintainer's Mac, with LANG
set to es_ES.UTF-8. Three spawns in i18n.test.js built their environment
inline from process.env and asserted on English output, so they inherited
whatever the machine said. A CI runner leaves LANG unset, so the bug was
invisible to the only place that was looking.
Five variants of that environment object had grown across the test files by then.
Three of them cleared LANG, LC_ALL and TRAZUM_LOCALE; none cleared
LC_MESSAGES, which detectLocale also reads — so even the files following the
"correct" pattern were one variable away from the same failure.
There is one environment now, in packages/cli/test/env.mjs, and it clears the
list of variables imported from the detector rather than a copy of it.
LOCALE_ENV_VARS is exported from packages/cli/src/i18n/index.ts and the
detector maps over it, so the list is the implementation: a variable added there
is read by the detector and cleared by the tests in the same commit, or in
neither.
It clears rather than pinning TRAZUM_LOCALE: 'en', which was the first attempt
and was wrong — that outranks the project config, so every test taking its
language from "locale": "es" in trazum.config.json would have reported in
English and asserted against the wrong catalogue. Clearing the environment leaves
the precedence chain intact and only removes the machine from it.
Two guards keep it: one fails when any test file builds a spawn environment inline, naming the file, and one asserts the shared environment clears every variable the detector reads. Both mutation-tested, including reverting a file to the exact shape that carried the bug.
Added#
The pull-request comment leads with what the branch costs. The Action has been posting a budget table since it shipped: a list of files and whether each one fits its ceiling. Useful, and not the question a reviewer is holding. What a pull request proposes is a change, so the comment now opens with the change — against the baseline recorded on the base branch — and puts the ceiling check underneath it.
> [!CAUTION]
> **This branch adds 64 tokens (+67.4%) to the prompts here** — over the limit of 0 tokens, 5%.
| | Prompt | Baseline | Now | Change |
|:--:|---|--:|--:|--:|
| 🆕 | prompts/triage.md | – | 64 | +64 |
Monthly cost **$129.75 → $132.95** (+$3.20)Only the directions that cost money are itemised. A list of every file that shrank buries the two rows somebody has to act on — though a branch that made things cheaper still gets its headline, because that is worth saying.
The rendered block comes from the same outcome the exit code was computed from, so a green comment and a red build cannot disagree. It shouts only when a threshold was actually crossed, names every limit that was crossed rather than the first, and refuses to print a monthly delta when the scenario or the price list moved — a figure in a pull-request comment gets quoted in a meeting, and two different measurements subtracted is not a saving.
No change to the Action was needed. It already posts whatever
check --markdown-out writes, so the report carrying the cost diff is the whole
mechanism.
Added#
trazum baseline, and a check that fails on drift rather than only on a
ceiling. budgets answers "does this file fit". That is a ceiling, and a
ceiling has a blind spot: a repository sitting at 95% of every budget passes
forever while a pull request adds four hundred tokens across a dozen files.
Nothing busted, bill up. A baseline answers the other question — did this get
worse than the commit we agreed on — and it is the twelfth command.
trazum baseline [dir] records what the estate costs now to a file you commit.
With a baseline block in trazum.config.json, trazum check on a directory
reads it and gates on it with no flag, because a gate you have to remember to
pass an argument to runs in the author's terminal and not in CI. --no-baseline
skips it for one run.
The behaviour the whole thing turns on: a prompt that is new counts. Comparing only the paths present in both documents would let a five-thousand-token addition through every threshold — it is in neither the grown list nor the baseline total — so the demonstration case is a run where every budget is green, no existing file grew, and the build fails anyway because somebody added a file.
The threshold is in tokens, and the money is only reported. A dollar figure comes from the token count, the usage scenario and the price list, and two of those move for reasons that have nothing to do with the prompts. A baseline holding dollars would fail a build the day a model was repriced, calling a price change a regression, and a gate that cries wolf is a gate somebody deletes. When the scenario or the price list has moved, the report says which one instead of subtracting two different measurements and presenting the difference as a saving.
A declared-but-missing or corrupt baseline fails the run. A gate the config
asked for and could not execute is not a pass; otherwise deleting one file
silently switches CI off. That includes a hand-edited totals.tokens that
disagrees with its own per-file counts — the corruption that otherwise looks
completely normal. Neither threshold has a default and omitting both is a config
error, because every default here is silently wrong: zero tolerance gets the
block deleted within a week, and a generous one is a gate passing things nobody
agreed to.
trazum.baseline.json's format joins the frozen API in VERSIONING.md, and it
is the strongest of those promises because the file is committed: it outlives the
Trazum that wrote it, so the document carries a version and an unknown one is a
loud error rather than a best-effort read.
Internally, the directory walk that decides what counts as a prompt was extracted
from checkDirectory so both commands share it. Two walks would be two
definitions of the estate, and the baseline would end up recording files the gate
never checks.
Changed#
Trazum was selling the weakest half of itself. The front page led with "cut what your prompts cost", and directly underneath it the demo — real output, not a mock-up — showed the rules recovering $24.00 a month while a single advisory sitting above them was worth $528.40. Twenty-two times more. The tool has been telling the truth about that gap for several releases; the pitch had not caught up.
The framing now leads with the finding rather than the trim. The headline is that most of an LLM bill is not the prompt, the advisories come first in What it actually does with the trimming after them, and the caption under the demo points at the two lines that make the argument instead of hoping the reader notices. The package descriptions, the web app's title, tagline and lede all say the same thing in both locales.
Nothing about the product changed — no rule, no advisory, no number. This is the description catching up with what was already being measured.
The two counts the pitch now rests on — thirteen advisories, twelve rules — are
asserted against RULES and the AdvisoryId union, so a fourteenth advisory
cannot ship while the front page still says thirteen.
Fixed#
The zero-dependency invariant was documented for three packages and enforced
for two. security.test.js looped over a typed list — packages/core and
packages/cli — while SECURITY.md credited the invariant to all three
published packages. packages/mcp hand-rolls its JSON-RPC layer for exactly
this reason, and the test named as the reason had never heard of it. The list is
derived from the root workspaces globs now, with an assertion that the
derivation found the MCP server at all, so a suite that quietly resolves to
nothing cannot report "0 failures" from having checked nothing.
CONTRIBUTING.md's first rule said "the core and the CLI" and now says what is
actually enforced, including why apps/web is exempt.
Nothing checked that the README's images exist. A moved or renamed asset
renders a broken-image placeholder to every visitor on the front page and tells
the person who moved it nothing. Local src and srcset paths are now asserted
to exist, absolute URLs deliberately excluded — somebody else's uptime is not
something a test here can hold.
The README's layout diagram also described apps/web as "Optimise and Compare",
one tab short since the prompt library shipped.
Added#
A real screenshot of the web app on the README, light and dark, swapped by
prefers-color-scheme so it matches the theme the reader is already in. Captured
from the production build rather than drawn, on the same wordy support prompt the
CLI demo uses.
Changed#
The web app got a display voice, two tiers of depth, and browser surfaces that belong to its own palette. It had been correct and characterless: shadcn's shapes wearing Trazum's colours, one system sans doing every job from the wordmark to the percentage that is the entire point of the page, and a dark theme in violet-grey borrowed from every other dashboard. Nothing about the identity changed — paper, terracotta and ink are exactly what they were — but the page now looks like it was built rather than assembled.
Fraunces Variable is self-hosted from npm and carries the wordmark and the
figures; the CSP's font-src 'self' holds, so no font is fetched from a foreign
host at runtime. Elevation is now two tiers and only two: working surfaces get a
hairline and a breath of shadow, and the result panel — the page's one focal
moment — drops its border rather than drawing the same edge twice. The dark
palette was warmed to the same umber cast as the light one so the terracotta
sits on Trazum's paper in both schemes. Selection, caret, focus ring and
scrollbars are themed from the palette, and the body sets tabular figures,
because every number on this page is a measurement and measurements align or
they wobble.
Two defects went with it. The reorder callouts were drawing a 3px coloured rule
down one side — a border doing a highlight's job — and are now a terracotta
wash. And on a narrow screen the history card sat between the Optimise button
and the answer the reader had just pressed it for; the two column wrappers
collapse to display: contents below lg, which makes every card a grid item
and lets history take order-last without moving a line of markup.
Added#
The README leads with the receipts, and every markdown file caught up with the
code. The front page now opens with a transcribed — not mocked — optimize run
as an SVG terminal, down to the closing line where Trazum admits the advisory is
worth 22× what the rules saved. The first draft of that SVG contained exactly one
invented figure, which its own header comment forbids; it was caught and replaced
with the real third advisory. A new guard derives the front page from the
workspace manifests: the architecture diagram had silently omitted @trazum/mcp
for the whole day that package existed, and now the next package added has to
appear in the README or publish.test.js fails.
The sweep also caught: SECURITY.md crediting zero dependencies to "the core and
the CLI" when three packages now carry the invariant; CONTRIBUTING.md counting
three workspaces and describing phrases.ts as "Spanish (and, in time, other)"
seven languages later; docs/releasing.md publishing two packages when the
workflow publishes three; the CLI README's command table listing five commands of
eleven; and VERSIONING.md freezing every API surface except the newest one —
the MCP server's tool names and input schemas are now part of the promise.
RELEASES.md and ROADMAP.md gained the account of everything that shipped
under 1.8.0's banner since their last update, provider layer and measurement
layer both.
Automatic recovery from container rollbacks, at scripts/recover-workspace.sh
and a Claude Code SessionStart hook. The remote environment this repository is
developed in restored its container disk to a stale snapshot more than twenty
times across two working sessions — every tracked file reverted, mid-work,
silently, always to the same commit. The first draft of this very script was
destroyed by the failure it exists to repair, one commit short of being safe.
A script inside the repository cannot prevent that, and this one does not claim to: it reverts along with everything else. What survives a rollback is the remote, so recovery is always fetch, reset to origin/main, reinstall — and the script makes those one safe move. The rollback signature is precise (HEAD strictly behind origin/main) and everything else is refused: a tree that is ahead is work in progress, a diverged tree is a choice no script should make (exit 1), and uncommitted changes are stashed by name before any reset rather than discarded.
.claude/hooks/session-start.sh runs it at the start of every Claude Code on the
web session — and only there, guarded on CLAUDE_CODE_REMOTE, because on a local
machine the tree is the developer's own and resets are not a hook's call.
Eleven behavioural tests drive the real script against real git repositories in
temp directories, including the one that matters most: the rollback also reverts
.git's remote-tracking refs, so a script that compared HEAD to origin/main's
ref would see them equal and announce nothing to recover. The fixture builds
exactly that state, and only a real fetch passes it. Thirteen mutants, thirteen
killed — among them "the stash disappears", "an ahead tree also gets reset" and
"a push appears", each the difference between a recovery script and a data-loss
tool with a reassuring name.
The definitive fix is platform-side — recreating the environment so it stops restoring a stale snapshot — and is not something a repository can do to itself.
Added#
trazum prune <file> --cases <file> — which few-shot examples earn their tokens,
measured rather than guessed. The eleventh command.
The redundant-examples advisory asks a textual question: does this example look like
an earlier one? This asks a stronger one: does removing it change any answer? Two
examples can be textually unalike and teach the same thing, and the few-shot section
is routinely most of a prompt.
Leave-one-out against the prompt's own noise floor. Ask the full prompt twice to
learn how much the model disagrees with itself, then remove one example and ask
again; a removal that moves the answer less than that did no observable work. The
thresholds come from evaluate's verdictFor rather than a second set.
The only command that asks before spending. The bill is (2 + examples) × cases
— 220 calls for a nine-example prompt over twenty cases. Without --yes it prints the
figure and stops, and it prints it before looking for a provider, so the cost is
visible without a key configured. plannedCalls is exported and pure, and a test
asserts the number it promises is the number spent.
It reports "no effect on these inputs" and never "delete this." An example may exist for a case the given inputs do not contain. Nothing is edited, and the strength of the claim is bounded by the inputs — which only the caller can judge.
withoutExample locates blocks by position rather than by
prompt.replace(text, ''): a copy-pasted few-shot section contains identical blocks,
and a text replace would match the first occurrence for both, so measuring the
removal of the second would describe the removal of the first.
Twelve mutants, twelve killed. Two defects came out of it that no test would have found, because both were about reading the output:
- The first draft duplicated
agreementfromevaluate.tsas a bag-of-words F1 while that one is Jaccard over normalised text — two different numbers under one name, with a comment in the copy claiming they were the same measure.agreementandpooledare now exported and shared, which makes the comment true. - The report put a green tick beside "0% agreement without it", which reads as
approval next to the one line meaning "leave this alone", and printed
Example:as every block's identifying line. Both only visible by running it against a local stand-in provider, which is how they were found.
@trazum/mcp — Trazum as an MCP server, so an agent can price and budget a
prompt before it sends it. Three tools over stdio: check_prompt,
optimize_prompt, list_models. It runs on the caller's machine, one process
spawned by the client exactly like the CLI — no service to host, and no prompt
leaves the machine.
check_prompt has three outcomes rather than two, which is the reason it exists:
inside budget, over budget but the rules would fix it, or over budget with content
that has to be cut. A boolean throws away the actionable half.
What it cannot do is the design. No paths — every tool takes text, and the
package imports @trazum/core rather than @trazum/core/node, so the file-reading
capability is absent rather than unused. No network: --suggest and eval are
deliberately not exposed, because a tool an agent can invoke in a loop must not be
able to spend the caller's money. No writes.
The JSON-RPC layer is written by hand, and that was not the first attempt. It
used @modelcontextprotocol/sdk and thirteen tests passed against a real process.
Then publish.test.js refused it: every publishable package here carries no runtime
dependencies outside the repository, and the reason security.test.js gives — every
dependency is somebody else's code running on untrusted text — applies to an MCP
server with more force than anywhere else in Trazum. Relaxing the invariant at the
point it matters most would have been backwards, so the invariant won.
What that costs is stated in the module: the implementation covers initialize,
notifications/initialized, tools/list, tools/call and ping, and answers
anything else with -32601. It has been driven by a raw newline-delimited client in
the tests, not by every MCP client in existence.
Writing it by hand immediately produced the bug that justifies testing it: the
notification check sat inside the method switch, listing the two notifications/*
methods by name. A notification is defined by the absence of an id, not by its
method, so {"jsonrpc":"2.0","method":"initialize"} with no id got a reply —
a protocol violation some clients tolerate and others hang on, which is the worst
kind because it works in testing. A test that asked for the rule rather than for the
two names found it.
Seventeen mutants: fifteen killed by tests, two by the compiler. Four new guards in
publish.test.js and the release workflow had to be updated too — a third
publishable package needs a README, a LICENSE, provenance, and a publish step
ordered after @trazum/core, and every one of those was a test failure rather than
something anybody remembered.
A .pre-commit-hooks.yaml, for teams who manage hooks with pre-commit.
scripts/pre-commit stays the recommended path; this is for the repositories that
already have a .pre-commit-config.yaml, which in practice means Python shops whose
prompts live in .py string literals.
Two hooks rather than one with a flag: trazum-check is a gate and fails the
commit, trazum-doctor never does. Collapsing them would let a --survey argument
silently turn a gate into a report.
It needs the first npm publish to work, and says so. The trazum executable
comes from additional_dependencies rather than from installing this repository,
and the two alternatives were tried rather than reasoned about: installing the repo
root gives Executable trazum not found, because the root is a private workspace
root with no bin; adding a bin plus a prepare that builds gives
Workspaces not supported for global packages, because pre-commit installs with
npm install -g.
The mechanism is verified with locally packed tarballs standing in for the registry — the gate fails a prompt over budget, passes one inside it, and the survey hook exits 0. The registry lookup is the only untested part.
An advisory for a schema the request could carry instead of the prompt. The one finding here that is not a trade-off.
A prompt that spells out its output shape in a fenced block pays for it in input
tokens on every call, and gets the weaker of the two available guarantees for the
money: prose asks the model to comply, a response schema makes the decoder comply.
output_config.format, response_format, responseSchema — every major API takes
the same shape as a request parameter. Moving it is cheaper and stricter.
Reported, never done. It is not a change to the prompt but to the code that
sends the prompt, and a rule that deleted the schema would leave a prompt asking
for a shape it no longer describes, sent by a client nobody updated — strictly
worse than the prompt it started from. A test asserts the schema and its fences
survive --level aggressive.
The one way it could do harm, and what stops it. A fenced JSON block is either
an output contract, which moves for free, or data a few-shot example needs, which
breaks the prompt if moved. Nothing guesses: a block counts only when a phrase from
the new OUTPUT_CUES_BY_LANGUAGE appears in the 240 characters before it. A schema
with no cue is left alone; a prompt in a language the dictionaries do not cover
raises nothing at all — a false negative, which is the right direction to be wrong
in, and stated as one rather than papered over.
The cue is matched through normalizeForCompare, so FORMATO DE SALIDA — and
formato de salida: are the same phrase, and it is quoted back verbatim from the
prompt rather than translated: it is the author's text, not the report's.
The figure is attached and the uncertainty is in the words. Trazum knows how
many tokens the block holds, which is reproducible; it cannot know from here
whether a given provider offers the parameter. Withholding the number for that
reason would be the wrong trade — it is right if the move is available — so the
advisory says plainly that it does not check. The same posture as
model-downgrade, which carries a figure and admits to being a keyword heuristic.
Thirteen mutants, thirteen killed. One found a real defect: the first draft
filtered out keys shorter than three characters, copied from the restated-format
detector where it stops a two-letter key matching a word in prose. Nothing is
matched against prose here, so all it did was undercount schemas whose fields are
called id or ok. Deleting it changed no test, which is what a line with no
reason looks like; it is gone, with a test for short field names in its place.
Added#
trazum doctor finds preambles that could share a cache entry and do not. The
first finding in this repository that no single prompt can produce.
Prompt caching is a byte-for-byte prefix match, so twelve prompts assembled from
the same system preamble — identical except that one has a trailing tab, another
reordered two bullets, and a third writes E-Commerce where the rest write
e-commerce — occupy twelve cache entries and share nothing. Every one of those
files is individually fine, which is exactly why no per-prompt analysis finds it.
drift says which kind of work it is: whitespace means the text already agrees
and a formatter fixes it; wording means somebody has to pick one.
Three refusals, and they are the design:
- Grouped by the first block. Caching matches from the start of the request,
so prompts whose opening paragraphs differ share nothing however identical the
rest is. Grouping on a later block would name prompts that can only be made to
share a prefix by reordering their instructions — the one transformation this
repository keeps out of
aggressivefor being dangerous. - Gated on the model's own cacheable minimum, via a new exported
cacheableMinimum. A model whose caching isunknown— what the live pricing overlay assigns to one it has never seen — yieldsInfinity, so nothing is reported. Telling somebody to unify a preamble across twelve files to enable caching their provider may not offer spends their afternoon, and unlike a wrong number on a report nothing later corrects it. The same directory against Haiku 4.5, minimum 4,096, produces nothing for a 1,398-token preamble. - Prompts already byte-identical are not reported. They share an entry today.
No dollar figure, and that is a finding rather than a gap. The saving lives in
the cache hit rate, and cacheHitRate is an input to the cost model rather than
something it derives — --cache-hit-rate applies one value to every prompt, so the
model has no term for how many distinct cache entries exist. Pricing this would
mean inventing how the calls are spread across the group, which is the one thing
here only the operator knows. A test asserts structurally that no field on the
result looks like money.
Thirteen mutants, eleven killed. The two survivors are equivalent rather than uncovered — two independent guards cover the same case, so no test can distinguish them — and both are documented in the test file so nobody removes one believing the other carries the weight.
Fixed#
A suite crashed on every clean checkout and exited 0. token-band.test.js read
fixtures/ with an unguarded readdirSync, and that directory does not exist until
somebody runs scripts/measure-token-band.mjs. So it threw ENOENT during suite
construction, node's runner printed the stack as a diagnostic, reported fail 0 and
exited 0 — which is precisely what the top of that file forbids: "'0 failures' from
a check that measured nothing is the most misleading thing a suite can report." The
skip beneath it was written for a directory that exists and holds no per-provider
file; it never covered the directory being absent, which is this repository's normal
state. Found while diagnosing an unrelated verify failure.
Fixed#
The README claimed prompts are never stored on any server. They are, once the
prompt library is switched on: trazum_prompt_versions.text holds the text of
every saved version, and has since the library shipped. The library is off by
default, so the sentence was true in the configuration everybody develops in and
false in the one an operator opts into.
That is the same shape as the Content-Security-Policy that blocked analytics nobody had enabled, found the same day — and worse in one respect. A broken policy eventually breaks visibly for the operator who enabled it. A privacy sentence is read once, by somebody deciding whether to trust the thing, and nothing ever tells them it was wrong.
The section now states both configurations and names the column. Three tests keep
it honest: the schema is asserted still to store prompt text, so if that ever
stops being true the claim can go back to being absolute and a failing test says
so rather than the guard going quiet; the exact sentence that was wrong may not
reappear; and both configurations have to be identified by the thing that selects
them. docs/accounts.md was already accurate and is now linked from here.
Five mutants, five killed, the first of them the original defect put back verbatim.
The new Content-Security-Policy blocked analytics, silently. connect-src 'self' shipped in the same change as the nonce, and Analytics.tsx posts to
https://eu.i.posthog.com. An operator setting NEXT_PUBLIC_POSTHOG_KEY got a
page that rendered perfectly and sent nothing, with the reason visible only in a
browser console nobody was reading.
Nothing caught it because the key is unset in CI and in development: the
configuration where it breaks is the one no test exercises. Found by reading the
policy next to the component while reviewing an unrelated posthog-js bump —
not by any check in this repository.
The host now comes from lib/analytics, which both files read, so the policy
and the request cannot name different hosts again. With no key the policy is
byte-for-byte what it was; with one, connect-src gains exactly one origin.
It is an origin, never the configured string. A policy is built by joining
text with ;, so a host of evil.test; script-src * would not have widened
connect-src — it would have appended a directive of somebody else's choosing.
new URL().origin discards everything a host source may not contain, and a
value that will not parse, or is not https, widens nothing at all.
Verified against a built server in both configurations: with the key set,
connect-src 'self' https://eu.i.posthog.com and nine of nine script tags still
nonced; without it, the previous policy unchanged. The badge keeps its own
default-src 'none'; sandbox either way. Nine mutants, nine killed.
CodeQL was one merge away from being permanently broken. Dependabot raised
github/codeql-action/init and github/codeql-action/analyze 3.37.6 → 4.37.6 as
two pull requests, because they are two sub-paths of one action and it treats
sub-paths independently. They are not independent: analyze reads the
configuration init wrote and refuses one written by a different version —
Loaded a configuration file for version '4.37.6', but running version '3.37.6'. Each pull request was red on its own for that reason, which is the
harmless failure. The harmful one is merging both halves in either order and
stopping halfway: the security job goes red and stays red, while every other
check on every later pull request is green.
Both are now bumped in one commit, and a test keeps them together — grouped by
owner/repo rather than by a list naming codeql-action, because the next
action split this way will not be that one. Verified against the real thing: the
test was run with the workflow put into each of the two pull requests' exact
states, and failed on both.
The bump was not optional maintenance either. v3 targets Node 20, which Actions has deprecated and is already force-running on Node 24.
Added#
A Content-Security-Policy with a real script-src. The web app had
frame-ancestors 'none' and nothing else — enough to stop clickjacking, and
nothing at all against script injection, so React's escaping was the only thing
between an XSS and full exploitation. It was documented as a limitation rather
than dressed up as a policy, and this is the limitation removed.
It needed middleware, because it needed a nonce. A policy worth having excludes
inline script, the App Router serves its flight data in inline <script> tags,
and a static header must therefore either allow 'unsafe-inline' — permitting
exactly the attack the policy exists to stop — or break the app. The value has
to differ per response, and a config header is one string for every response.
default-src 'self', script-src with a per-request nonce plus
strict-dynamic, connect-src 'self' to close the exfiltration channel,
base-uri, object-src, form-action. 'unsafe-eval' only outside
production. style-src keeps 'unsafe-inline', which is the one concession and
the cheap one: a stylesheet cannot execute.
Verified against a built server rather than asserted. Nine of nine script
tags carrying the nonce from the header, a different nonce on every request, the
page rendering, and /badge/<token> keeping its own default-src 'none'; sandbox — tighter than the site policy, and excluded from the matcher rather
than trusted to win, because this repository already shipped one change that
silently replaced it with something looser.
Thirteen mutants, all killed — the last one only after the test was fixed. It
asserted headers.set('content-security-policy', which
response.headers.set(…) two lines below also satisfies, so deleting the
request header left it green. Deleting it is not cosmetic: measured on a
built server, nine script tags and zero nonces. Next reads the policy off the
request to learn which nonce to stamp, so without that line the header is
perfect and the page is dead.
Two of the new tests were also caught matching their own comments — once on
default-src 'none' from the paragraph about the badge, once on randomUUID
from the sentence explaining why it is not used. Both now read the source with
comments stripped, which is the third time this repository has needed that.
Added#
Bedrock and Vertex, with their credentials signed by hand. The last two
providers whose auth is not a bearer token, and the reason there is no SDK here:
@trazum/core has zero runtime dependencies and a test that fails the build if
one appears, because every dependency is somebody else's code reading your
prompts. The AWS and Google SDKs are two hundred packages between them to
authenticate one request. SigV4 and the service-account JWT are about three
hundred lines, on WebCrypto so the browser-safe entry point stays browser-safe.
Bedrock goes through Converse, not InvokeModel. That is what makes it one
provider instead of six: InvokeModel takes each model family's own body shape —
Anthropic's messages, Meta's prompt, Amazon's inputText — so supporting
"Bedrock" through it means a 400 for every model nobody thought about.
Vertex caches its access token. A token lasts an hour and --suggest over a
directory makes one call per prompt; without the cache, forty prompts are eighty
requests, half of them to an endpoint that rate-limits. One cache per provider
instance, so two providers in one process cannot leak a token into each other's
requests.
Google's three HTTP-200 failures are now read by one shared function rather than two copies. Two copies of "is this answer complete" is one copy too many when the whole point is that a truncated rewrite reads like a finished one — and the error names Vertex or Gemini, because those are different consoles.
Neither has been exercised against the real service, and neither has the OpenRouter feed or the Gemini endpoint. This environment's network policy denies all of them. What the tests prove is stated in the files themselves: the shape is right, and the first real call is what proves it works.
CodeQL caught a weak assertion, and the weak assertion was hiding a bug.
Three host checks in the new tests used unanchored regexes, so
/oauth2\.googleapis\.com/ would also have matched
https://evil.example/?x=oauth2.googleapis.com. Not a vulnerability — the URL is
one this code built — but an assertion that would pass against a request to the
wrong host is not testing what it names. Rewritten to compare parsed hosts and
paths.
Asserting the path exactly is what then surfaced the real defect: Bedrock model
ids contain a colon, AWS's own URLs carry it unencoded, and the comment in the
provider claimed encodeURIComponent leaves it alone. It does not — it produces
%3A. The signature matched either way, because the same string is signed and
sent, so nothing else in the suite could have noticed; the request would simply
have gone to a path AWS does not document. The colon is preserved now and the
slash is not, because a slash in a path is a new segment and a provisioned-model
ARN contains both.
Twenty-eight mutants: twenty-six killed, two documented equivalents. Three of the
kills only became possible after the tests got better — deleting the region from
the SigV4 key chain, deleting the service, and dropping the AWS4 prefix from the
secret all left every test green, because the region and service also appear in
the credential scope inside the string to sign. "Different region, different
signature" is true of a signer whose key derivation is entirely wrong. The chain
is now derived a second time from the specification and compared.
The two equivalents are the header sort: removing it changes nothing while the literal list is already in order, and reordering the literal changes nothing while the sort is there. Doing both at once is killed, which is what shows the ordering is actually asserted.
Added#
A native Gemini provider, and the reason it needs one while eleven other providers do not.
openai in TRAZUM_LLM_PROVIDER is not "OpenAI", it is the wire format — so
OpenRouter, LiteLLM, Groq, Together, Fireworks, DeepInfra, DeepSeek, Mistral,
Cerebras, SiliconFlow, Ollama and vLLM are all a base URL away and always were.
The README now names each one with its exact URL, because true and undocumented
is not much better than false.
Google's API is a different document: the system prompt is systemInstruction
rather than a turn, and the answer is a candidate's parts. What actually earns
it a function is that three of its failure modes arrive as HTTP 200 — a
blocked prompt (promptFeedback.blockReason, with no candidates at all), a
truncated answer (finishReason: MAX_TOKENS), and a candidate with no text. A
client that checks res.ok treats all three as success, and the second is the
worst thing that can happen to a rewrite pass: half an answer reads exactly like
a whole one. All three throw.
The key goes in x-goog-api-key, not the ?key= that Google's own examples
use, which writes a live credential into every proxy log and Referer between
here and there. The model name is escaped into the path, and the endpoint goes
through the same gate as every other outbound call.
Eleven mutants, all killed — one of them by the type checker, which is a real kill: removing the empty-text guard stops the function compiling because the return type stops being a string.
Added#
--pricing-live: prices from OpenRouter instead of a table somebody typed.
The bundled catalogue is stale the day after it is written and only ever covered
the providers whoever wrote it reached for — so a user on Groq or Together got no
figure at all, from a tool whose entire output is figures. OpenRouter publishes
price and context window for hundreds of models across dozens of providers, as
data, at a URL.
Opt-in, because it is a network call: rule 1 is that no feature makes one a
prerequisite for optimising a prompt. The CLI fetches and hands the core a
value; openrouterOverlay is a pure transformation, so it is testable without a
network. Through checkedEndpoint and SAFE_FETCH_INIT like every other
outbound call here, so redirects are refused rather than followed to the
metadata network. A --pricing file still wins: somebody who wrote prices down
meant them.
What that feed does not publish, and what is done about it. It has no opinion on whether a model has prompt caching or the minimum prefix it caches at — and that is the input to the largest saving Trazum reports, an order of magnitude above what the trimming rules recover.
So CachingMode gains unknown and cacheMinTokens becomes nullable, and a
model that arrives from the feed carries both. The caching advisory declines,
and trazum models prints a dash rather than a zero. The two available lies are
symmetrical and both worse than silence: claim caching works and Trazum offers a
saving nobody can buy at any price — the Mistral bug in a new costume — and
claim it does not and Trazum hides the biggest saving there is.
Capability and tier gain unknown for the same reason, so a model whose
capability nobody recorded is neither recommended to somebody else nor told it
is overpowered. For a model the bundled catalogue already has, only price in,
price out and context window are refreshed: the rest was written by somebody who
looked it up, and replacing a researched fact with a blank is not a refresh.
Fixed#
An added model could carry no capability at all. applyPricingOverlay
required six fields for a model the bundled catalogue does not have, and
capability — a required field of ModelPricing — was not among them, so
as ModelPricing produced an object the type says cannot exist. Now required,
which is why two overlay fixtures needed a line.
Ten mutants over the new code; nine killed and one documented equivalent. Two of the kills were bugs of mine rather than of the tests:
- The unknown-capability guard was written as an early
return, which skipped every advisory after the model check — output-dominated, contradictory instructions — none of which has anything to do with a model's tier. TIER_ORDER.unknownwas set above every real tier, which makes the downgrade comparison true for every prompt; the only thing then standing between that and a recommendation was an unrelated provider filter. Two accidents covering for each other is not a design. It is-Infinitynow, so the ordering carries the rule and the guard beside it is deliberate redundancy — that is the equivalent mutant.
Fixed#
The token-band measurement could never have passed, and nobody could know.
measure-token-band.mjs hashed the corpus with NUL separators and
token-band.test.js hashed it with spaces, so the two digests could not match.
The first real measurement would have failed its freshness check with "the
corpus changed since it was measured — re-run scripts/measure-token-band.mjs"
— advice that produces the same failure however many times it is followed.
It went unnoticed because running the script costs an API key nobody had spent. The one workflow that discharges this project's central claim had never been executed end to end, and the check guarding it was broken in the way that surfaces only the first time it matters.
Fixed structurally rather than by making the copies agree, since two copies
agreeing is the state it was in when it broke: scripts/corpus-digest.mjs is
the single implementation and both sides import it. A guard asserts neither has
grown a second one — and it builds its own needle at runtime, because written as
a literal the assertion matches its own source and fails on the file it defends.
And a fourth raw control byte in a source file. Writing that shared module
put a real NUL into it on the first attempt, exactly as happened in
reorder-properties.test.js, github.ts and measure-token-band.mjs before
it. Caught this time by checking the bytes rather than the diff. The separator
is written as an escape.
Added#
measure-token-band.mjs --provider deepseek, and the distinction that makes
it safe to have.
±15% is the estimator's accuracy against Claude's tokenizer — the family it
was calibrated on, and the one every published claim refers to. Trazum prices
seven providers with that one estimator, and how far off it is on the others is
a question ROADMAP.md has open with a decision resting on it:
within 5% across families and a real tokenizer dependency is not worth taking;
40% out and it is.
So each provider writes its own fixture and only Anthropic's governs the published band. A cross-family fixture asserts corpus freshness and coverage, prints the error per text type, and asserts nothing about a band it was never calibrated for — with a guard that it cannot claim otherwise. Reading a DeepSeek number as the published band would be the same class of error as calling a release published because a changelog heading exists.
Two things the script now says out loud before sending anything: DeepSeek has no
free counting endpoint, so every sample is a real completion with
max_tokens: 1 and the prompt half is billed; and a run against it does not
discharge the ±15%.
Fixed#
blame reported a git it could not run as a repository with no history.
git() collapsed every failure into null — git missing, git exiting non-zero,
and the process failing to start at all — and revisionsFor turned null
into []. So a fork the kernel refused with EAGAIN, which is a fact about the
machine for one instant, reached the author as git has no commits touching p.txt: a confident claim about their repository, made without having asked it
anything.
That is the shape of #58 — zero
rows, exit 0, once on CI, never reproducible — and it is the shape that cannot
be diagnosed afterwards, because its output is identical to the true answer.
Failing to run git now throws GitUnavailableError; an empty history still
returns an empty list. Transient spawn failures (EAGAIN, ENOMEM) are retried
once, bounded by the loop rather than by a condition inside it.
This does not prove EAGAIN caused that CI failure. Nobody knows, and the
issue was honest about it. What changed is that this failure can no longer
disguise itself as an empty history.
Eight mutants, all killed, and two of them were the tests being wrong rather
than the code. The first version of the regression test drove the CLI with
PATH stripped and passed against every mutant including the bug restored,
because blame checks gitAvailable before asking for revisions — the process
never reached the code under test. The second was worse: mutating the retry into
an unbounded loop did not surface as a surviving mutant, it surfaced as the
suite hanging until the runner killed it, which in CI is a job that burns its
whole timeout instead of failing in a second. The loop is now bounded by
construction.
A broken anchor in the README, and nothing looking for one.
#reordering-for-the-cache-reorder pointed at a heading whose real slug has
three hyphens, because GitHub turns cache: --reorder into cache---reorder. A
dead anchor renders as ordinary text and silently does nothing, so no other
check could see it. All 25 in-page links are now verified.
Documents that had gone out of date with the release. RELEASES.md claimed
1.8.0 was on npm and installable; nothing is published, there is no tag, and
npm view @trazum/core returns 404. The guard that exists to catch exactly this
missed it because it read "is there a ## X.Y.Z heading in the changelog",
which is a release cut in this repository, not a package on a registry.
Preparing 1.8.0 satisfied it and switched the assertion off. It now asks git for
a tag, which is what release.yml actually triggers on.
The same claim is now checked in both directions across RELEASES.md and both
package READMEs — the ones that open with npm install @trazum/… and are the
npm page. Untagged, the notice is required; tagged, it is forbidden. A note that
has to be removed by hand at release time is a note that survives three
releases.
ROADMAP.md and docs/releasing.md follow: the release environment exists
and has been exercised by a dry run, the first publish has to be made by hand
because a trusted publisher is configured on a package page that does not exist
until the package does, the manifest count is four rather than three, and the
action-pin step describes the guard as it now works.
Changed#
The README is navigable. 1,431 lines with no way in: five badges, a table of the ten commands and what each answers, and a contents list. Nothing was removed.