Fixed#
Two
--jsondocuments carried the prompt text this file promises they never do.docs/json-output.mdopens by promising that "nothing here carries a session key or prompt text".trazum route --jsonprinted the wholeEvalReport, whosecases[]holdsinput— the caller's own case text — plusbaseline[0],baseline[1]andoptimized, three model answers.trazum prune --jsonprintedExampleContribution.text, which is a few-shot example and therefore prompt text by any reading. Neither carried aschemaVersion.Nothing said so, and the guard that enforces the promise explains why.
format-promises.test.jsruns it over "every document this package can build — the profile, the roll-up and the prompt draft". Both of these are built in the CLI, so they sat outside a guard whose own comment describes the state they were in: a promise that quietly stops being true, because nothing would say so.The measurement survives; only the strings go. Both documents keep their per-item shape, because the similarity scores are the evidence behind the verdict and
indexsays which example a verdict is about. Anyone who wants to read the answers has the human-readable output, which never left the terminal.routeDocumentandpruneDocumentare named functions now rather than inline objects, which is what makes the guard possible at all: reaching either document for real costs provider calls and an API key, so a test that demanded a live run would never run in CI.Three plants, three failures, including the one that matters in the other direction: returning a nearly empty document passes a stripping guard perfectly, and fails
the per-example evidence went with the text.Named and not built: five commands emit
--jsonwith no contract table indocs/json-output.mdat all —route,prune,store,conformandwatch.contract-coverage.test.jschecks documented-against-tested in both directions and cannot see a document nobody ever wrote down. The guard that derives the list fromCOMMAND_FLAGSwas written and set aside: it fails until those five tables exist, and writing them is its own piece of work.
Added#
trazum from-heliconereads a Helicone request export as a usage log. The fourth converter in the patternfrom-claude-codestarted: a pure function turns one tool's export into Trazum's records and every door prices it from there. Helicone sits as a proxy in front of the provider and keeps every request it saw, so a team using it already has the export and needed no new instrumentation to answer where did the money go.The format is derived, not remembered. The columns are the SELECT that builds Helicone's own request table —
web/lib/api/request/request.tsin Helicone/helicone — and the shape itsPOST /v1/request/queryendpoint documents. A converter written from memory of an API is a converter that silently mis-reads somebody's bill.It takes three columns to answer what model ran.
request_model,model_overrideandresponse_modelcan disagree, because the override exists precisely so a proxy can send a different model than the caller asked for. The response wins, then the override, then the request — a bill is about what was billed, not what was intended — andmodelDisagreementscounts the rows where they differed, so a substitution is a fact a reader sees rather than one they discover inside a total.What does not cross. A Helicone row carries the request body, the response body and
request_user_id, which is an email address in Helicone's own documented example. None of it is read: the converter names the fields it takes and takes nothing else, and a fixture plants one marker in each and greps the whole conversion for it.What it refuses to invent.
cache_enabledis a flag with no token split behind it, so a converted record carries no cache fields and the caching questions come backcannot-tell— the same refusalfrom-otelandfrom-litellmmake. Andrequest_idis one call, not a conversation, so the records carry nosessionand the conversation findings stay unavailable rather than being answered from a per-call id, which would report every call as a conversation of one.looksLikeHeliconeis deliberately narrow, and the plant that proves it is a LiteLLM spend log and an OpenAI usage response: both carryprompt_tokens, and a detector loose enough to claim either would take somebody's export away from the converter that understands it.The tier heuristic refuses when the prompt argues with itself. The score is
complex × 2 − simple × 2, so a prompt carrying four signals of a hard task and three of an easy one cancels tosonnet— the identical answer to a prompt with no signals at all. Those are opposite situations reported with one number: one is "nothing here suggests a tier", the other is "this prompt contradicts itself about which tier it needs", and only the second is worth saying out loud.A new
tier-signals-conflictadvisory says it, andmodel-downgradedoes not fire while it holds. That order matters: a saving computed from a tier the heuristic cannot stand behind is a dollar figure with nothing underneath.The threshold is derived from the weights, not from taste. A lead of one signal or fewer counts as conflict, because each signal moves the score by 2 and prompt length moves it by up to 2 on its own: a one-signal lead is inside what length alone contributes, and a lead of two or more is a majority the size term cannot manufacture.
recommendTierDetailedreports the counts and the verdict;recommendTierkeeps its shape, because the 1.x line does not change one.Four plants, four failures, including the one that matters most in the other direction: making the refusal unconditional fails
still recommends a tier when one side clearly leads, so the guard cannot be satisfied by refusing everything.
Changed#
The published
±10%token band was a fit over its own calibration set, and there is no single band any more. For eight releases every report said the same figure about every prompt. It was measured over a corpus of twenty-one samples that held thirteen files of Latin prose and exactly one each of code, numeric and punctuation — and those single files were the set the estimator's constants had been fitted to. So the number was not a measurement of the estimator. It was a measurement of its own training set.Twenty-six ordinary samples broke it. Measured against Anthropic's counting endpoint, the same estimator that is 5.6% out on prose is 25.1% out on a run of identifiers and 32.5% out on a CSV ledger. Telling somebody ±10% about their ledger was telling them a number that is wrong about their prompt specifically, and every dollar figure below it inherited the error.
The band is a property of the text now.
bandFor(text)in the newband.tsanswers with one of four measured figures, and every surface prints it: the report's estimate line, the check and blame markdown, the MCP tool output, and the two advisories that hedge near a threshold.Kind of text Band Worst sample Samples CJK, all three scripts ±4% 3.2% ( cjk-japanese-technical)6 Latin prose and few-shot blocks ±6% 5.6% ( numeric-matrix)18 Code, markup and quoting ±26% 25.1% ( numeric-identifiers)16 Digit-dominant tables and ledgers ±33% 32.5% ( numeric-tabular)7 It deliberately does not classify by text type, and that is a finding rather than a shortcut. Measured by character mix, code and punctuation overlap completely —
code-sqlis 7.7% symbols andpunctuation-markupis 17.5%, withcode-heavyat 15.6% sitting between them — and two of the three few-shot samples are indistinguishable from prose. A classifier over those would be a guess wearing a measurement's name, and it would hand out the wrong band precisely where the band matters most. So the buckets are the separations the corpus actually offers, and where two classes overlap the worse band wins: an overstated uncertainty costs a reader some confidence, an understated one tells them a wrong number and lets them act on it.The thresholds are the widest gaps the corpus offers, not round numbers somebody liked. CJK at 0.5 — the CJK samples are 90.1% to 94.5% CJK and every other sample is 0.0%, with nothing in between. Digits at 0.30 — the two purely numeric samples are 47.7% and 49.7% digits and the next highest anywhere is 15.3%. Symbols at 0.07, and this one is stated as thin:
code-sqlat 7.7% againstportuguese-proseat 6.1% is 1.6 points of margin. Thin in the safe direction, because a prose sample that crosses it is given a wider band than it needs and no sample is ever given one narrower than its own error.ESTIMATE_ERROR_BAND_PCTis now 33 rather than 10, and it is only for callers that must print one figure without holding the text — a figure covering the whole catalogue of text can only be the worst of it. Widening it makes every claim that reads it true where it had been false, at the cost of understating the estimator on prose. A caller with no text cannot know which it is holding, and there is one safe direction to guess in.The guard is planted rather than asserted.
token-band.test.jsholds every sample against the bandbandForgives it, requires no bucket's band to be narrower than its own worst sample, and requires every bucket to carry at least one measurement. It cannot be satisfied by tuning a threshold: a sample sorted into a friendlier bucket gets a smaller band and fails. The single-source guard was extended too — it now reads a±N%in any live file and fails unless the code publishes that exact number, with source comments stripped first so a file explaining what the band was is not forced to falsify its own history.The estimator's error against a foreign tokenizer is said out loud, with the number. The report already knew to name the provider when the model was not Anthropic's. What it could not say is how far off — and a hedge with no figure is a hedge a reader discounts. The same 47 samples run against DeepSeek's own counter come out 94.5% wrong at worst and against Mistral's 103.1%, so those two families now get the measurement on the line:
26 → 21 -19.2% (estimated: the counter is calibrated on Claude, not DeepSeek V3, where it has measured up to 94.5% out)A family nobody has run gets "nobody has measured how far off it is there" instead, because
measuredForeignErroranswers null rather than the nearest number. Both halves are held: one test recomputes each published figure from that provider's fixture and fails on drift, another fails ifband.tsever states an error for a family with no fixture behind it at all.The corpus is 47 samples in ten languages, and the harness reaches Mistral. Twenty-six were added: seven code and markup samples, three few-shot, four more CJK including the first two Korean ones, and twelve purely numeric. Mistral joins Anthropic, DeepSeek, OpenAI and Google as a family the harness can measure — through the gateway's upstream allowlist, because
trusted-hosts.test.jsrefused to let a measuring script be a side door, and it was right to.The harness also retries a rate limit or a 5xx with doubling backoff and paces paid providers, after a 43-sample run died three times on the same endpoint.
--exact-tokensand the default measurement model. The band script's default Anthropic model wasclaude-opus-4-1, which is retired: the first run anybody had a key for returned a 404 from a model id nobody had checked since the catalogue moved. It isclaude-opus-5now.model-downgradenames the command that settles it. It said "measure the difference with your own evaluations", which is advice nobody can follow without already knowing how. It now namestrazum route <log.jsonl> --prompt-file <prompt> --cases <cases> --yes, which is the command that sends the same cases to both models and reports whether the cheaper one still does the job. The refusal above names it too, because a refusal that does not say what would settle the question is only half of one.
Fixed#
A character range that looked right sorted astral emoji as CJK.
band.tswrote its CJK test as five ranges of literal characters, and the last pair meant "compatibility ideographs, U+F900 to U+FAFF". Its opening character was U+8C48 — the ordinary unified ideograph that shares the glyph, not the compatibility one — so the range ran from U+8C48 to U+FAFF and took in the Yi syllables, the private-use area and both surrogate halves on the way.A page of emoji therefore came out
cjkand was handed ±4%: the narrowest band in the file, about text nothing has ever measured. It passed every assertion intoken-band.test.jsbecause no corpus sample lives in any of those blocks. CodeQL found it on the pull request that introduced it, in the same release whose entire subject is a figure that looked measured and was not.The ranges are numeric escapes now, and a test plants one character from each block that used to be caught — with the silent half checked too, so a guard that rejects everything cannot pass it.
Hangul was charged a placeholder nothing had measured. Every CJK character that was not kana was billed han's 1.05 tokens, and the corpus had no Korean in it to say otherwise. Two Korean samples in different registers — a support script and a technical note — agree in lockstep across the whole search: −10.6% and −10.0% at 1.20 tokens per character, −3.5% and −3.1% at 1.30, 0.0% and 0.0% at 1.35, and +3.2% and +3.4% at 1.40.
HANGUL_TOKENS_PER_CHARis 1.35, the CJK bucket's worst error went from 10.6% to 3.2%, and its standard deviation from 4.0 points to 1.0.This is the only calibration in the release that succeeded, and the reason it is trusted is that two independent samples moved together at every candidate rather than one sample landing on a number.
Punctuation was charged half a token each and costs closer to one. The divisor was
n / 2, fitted against a single punctuation-heavy sample. Five symbol-dense samples added this release all undercounted, andn / 1takes them in. The digit divisor was deliberately left alone: no single constant serves both numeric samples, and moving it to fit one would be the same fault as the band itself.Two hypotheses were tested and rejected, which is what the larger corpus bought. Digit-run length does not predict the numeric error —
numeric-tabularaverages 2.79 digits per run and is +32.5% out,numeric-heavyaverages 3.13 and is −5.0% — and neither does grouped-number density:numeric-versionsat 0.44 is −11.5%,numeric-tabularat 0.60 is +32.5%. With two samples each would have been fitted and shipped as a fix.The caching advisory quoted a threshold that was true of no model. It said, for every model in the catalogue, "a cache write costs 125% of the input price and a read costs 10%. Below roughly a 28% hit rate you pay more than you save." Two of those numbers were Anthropic's multipliers stated as universal. The third was not derivable from anything: break-even is where a cached token costs what an uncached one does,
h·read + (1−h)·write = 1, soh = (1 − write) / (read − write), and at 1.25 and 0.1 that is 21.74%.The advice was actively wrong for eight of the eighteen models.
gpt-5,gpt-5-mini,gpt-5-nano,gemini-2.5-pro,gemini-2.5-flash,kimi-k2,deepseek-v3andgrok-4write at 1× input price, so caching cannot lose money at any hit rate:(1 − 1) / (r − 1)is zero, which is the absence of a threshold rather than a small one. All eight were being told to consider turning caching off. The message now takes the model's own multipliers and says the different thing that is true of them.The multipliers were in hand at the call site and already passed to the advisory next door, whose comment says why a global constant is "an invented saving rather than an imprecise one". Only this message invented its own. The guard recomputes the threshold from the catalogue rather than typing it in, and one plant fired on the number while a second, matched against the wording of the replacement rather than the claim, did not: the sentence it was written to catch reads
Below **roughly** a 28%andBelow adoes not match it. It is matched onyou pay more than you savenow, which is the claim itself.A measured cache hit rate rose the more the cache was rewritten.
cacheReadSharewasreads / (input + reads), leaving cache writes out of the total it claims to be a share of. With 100 read tokens, no plain input and 9,900 written, it answered 100% where the truth is 1%.That is the worst direction for this number to be wrong in. It is handed to
optimizeascacheHitRate, which decides whether caching is paying off, so the workload burning money on writes was the one most likely to be told its cache was working perfectly. Writes are input tokens: billed at the input rate times the write multiplier, arriving in the sameusageblock.The downgrade advisory halved candidates that have no batch API. The saving multiplied the cheaper model by a hardcoded
0.5whenever the caller saidbatchEligible, while the current model's cost twenty lines above already usedrates.batch ?? 1. Three models carrybatch: null—kimi-k2,deepseek-v3andgrok-4— meaning no batch API at all, so the advisory offered money that cannot be bought at any price, made the downgrade look twice as good as it is, and could turn a negative saving into a reported one.The MCP registry read-back failed 1.81.0's release while the publish had succeeded. The listing job's last step asks the registry whether it now serves what was just sent. It reported
the registry serves '1.80.1' for io.github.Davmunrey/trazum, expected '1.81.0'seconds after the publisher printed✓ Successfully published — version 1.81.0. Both statements were true, and the check was wrong twice over.It took the first entry the search returned. The registry keeps every version of a server, so
find((entry) => entry.server?.name === name)answers "some version is listed" while the step reads it as "this version is listed". After 1.81.0 published, the search returned 1.80.1 first and 1.81.0 second. Verified against the live registry rather than reasoned about: the old expression reads1.80.1today, and would have failed every release from here on, not only this one. It now collects every version under that name and asks whether the wanted one is among them, so the answer cannot depend on ordering. When it is absent the error names what the registry does hold.And it asked once, 0.6 seconds after the publisher returned. The publisher confirms the write; the search index catches up afterwards.
scripts/mcp-registry-preflight.mjsalready learned exactly this about npm and polls until the answer arrives, and the lesson had not travelled one step further down. The read-back now asks for up to a minute.Nothing was actually broken by this:
@trazum/core,@trazum/cliand@trazum/mcpare live on npm at 1.81.0, the tag exists, and the registry listsio.github.Davmunrey/trazumat 1.81.0 with statusactive. The failure was a check reporting a broken release that was not broken, which is the worst direction for a check to be wrong in: it teaches whoever sees it to stop believing the red.Two plants, two failures: restoring the first-match expression fails with
the read-back takes the first listing by name, and removing the polling fails withthe read-back asks once rather than waiting for the registry to catch up.