Measured: a model you own against Claude Opus 5

Two tasks, measured the same way against the same frontier model. One reads fields out of long documents; the other routes short customer messages to one of 77 queues. They have nothing in common except the property that matters: a repetitive judgement with a history of right answers behind it.

They are on one page because the results are not the same shape, and a reader deciding whether this applies to their work needs to see both. On document extraction the frontier model is weak and the gap is enormous. On routing it is strong, and the gap is smaller but still decisive. Where the argument changes is cost, and that is set out under Cost rather than averaged away.

Benchmark one: contract field extraction

76 held-out documents, measured 2026-08-03. A fine-tuned model you own scored 1.34× the accuracy of Claude Opus 5 given the same worked examples, produced no malformed output at all, and cost 22× to 62× less per document at the providers' standard rates. What changes if you optimise that bill.

What was measured

• Task: ten fields pulled from commercial contracts — parties, dates, governing law, notice period, renewal and exclusivity flags, restricted activities. Public dataset, so the comparison can be checked rather than taken on trust.
• The comparison: Claude Opus 5 against a fine-tuned open-weight model, on the same 76 documents neither had seen.
• Twice for the frontier model. Once cold, and once with worked examples in the prompt — because the fair question is not whether fine-tuning beats an untuned prompt, it is whether it beats a frontier model given every reasonable advantage.
• Scoring: field by field. A wrong value counts against the model twice, as both a false positive and a false negative. Agreeing that a field is absent earns nothing. Malformed output is tracked separately rather than quietly discarded — the scoring rules in full.

Results

Seventy-six documents is a small sample, so the difference was tested rather than asserted. Resampling the documents five thousand times puts the gap at +0.129 accuracy, with a 95% interval of +0.077 to +0.179. The fine-tuned model came out ahead in every single resample, and the two ranges do not overlap. This is not a result that turns on which documents happened to be in the test set.

Why it wins

A frontier model arrives knowing everything in general and nothing about your business. It does not know which of two parties your team calls the first one, or that a silent contract means empty rather than a best guess. Those conventions live in your existing records, and a model trained on them absorbs them.

That is not a scoring trick. It is the entire mechanism, and it is the reason the advantage transfers to your task rather than being an artefact of ours.

It also shows up exactly where the theory says it should. The three largest gains are all judgement calls rather than lookups: auto-renewal 0.347 to 0.671, restricted activities 0.198 to 0.483, and exclusivity 0.547 to 0.789. On agreement type, which is close to a lookup and which a general model already handles well, the gain is 0.018 — almost nothing. The advantage is concentrated in the fields where knowing your conventions is what the work actually is.

Prompting narrows the gap. It also triples the bill.

The obvious objection is that the frontier model was never given a fair chance, so it was given three worked examples. It improved — from 0.314 to 0.383.

And the price per thousand documents went from $22.19 to $63.18. The examples travel with every request and are charged for on every request, for as long as you run the task. A fine-tuned model absorbs them once.

That is the case for owning the model, stated as arithmetic rather than opinion: you can pay to teach a frontier model your conventions on every single call, forever, or you can pay once and own the result. Prompting also produced more malformed output, not less — 6 broken records out of 76 rather than 4.

Held against that prompted baseline, the fine-tuned model is 1.34× as accurate and 62× cheaper per document. Held against the cold baseline, which is cheaper to run but less accurate, it is 1.63× as accurate and 22× cheaper. Both comparisons are on this page because picking whichever one flattered us more would be the easiest thing on it to catch.

Field by field

The fine-tuned model leads on seven of the eight fields the test set can actually measure, and the largest gains are on the judgement calls — exclusivity, auto-renewal, restricted activities — where a general model has to guess at what your records mean by them.

Governing law is the exception, and it is worth being precise about why: this dataset records it as a sentence of quoted text rather than a value, so every model is scored on matching an annotator's phrasing. At 0.076 against 0.044, neither reads the clause reliably. It is a field-definition problem, and it is the kind of thing an audit finds before anyone builds on it. Two further fields are excluded entirely — the test set contains one example of one and none of the other, and an accuracy computed over a single example is noise rather than evidence.

When it is unsure, it says nothing

On effective dates, across the documents that have one, the fine-tuned model was correct or silent — it never returned a date that contradicted the source. The frontier model answered more often, and some of those answers were wrong.

For document work that difference matters more than the headline. An empty field is caught by a reviewer in a second. A confident, plausible, wrong date is not caught at all — it propagates into whatever the record feeds. A model trained on your records inherits your conventions about when to leave a field alone. It is also why the malformed-output rate is zero: the model produces a valid record every time, so nothing downstream has to defend against broken input.

Benchmark two: routing customer messages

600 held-out customer-service messages, each routed to one of 77 request types, measured 2026-08-04. A fine-tuned model you own scored 0.9217 against Claude Opus 5's 0.8296 and again produced no malformed output at all. Stated the way it will actually be felt: the error rate falls from 17.0% to 7.8% — 54% fewer mistakes.

Why the multiple is the wrong way to read this one. The gap is 1.11×, which sounds marginal next to the 1.34× on contracts. It is not: accuracy is capped at 1.0, so multiples compress as a task gets easier, and the contract figure comes off a much weaker baseline. What a business actually feels is the mistakes, and there are 54% fewer of them.

Resampling the 600 messages five thousand times puts the gap at +0.0921, with a 95% interval of +0.0645 to +0.1195. The fine-tuned model came out ahead in every single resample, and the two ranges do not overlap at all. This gap is narrower than the contract one, so the interval matters more here, not less — it is the difference between a result and a coincidence.

What was measured

• Task: a public dataset of real customer-service messages for a bank, each labelled with one of 77 request types — card arrival, a disputed fee, a failed transfer, and so on. Public, so the comparison can be checked rather than taken on trust.
• Both models were given the full list of 77 types in the prompt. Without it the frontier model would be guessing at exact wording rather than at meaning, which would measure the wrong thing.
• The dataset's own train/test split was kept. Re-drawing it would have put the official test messages into training and produced a much better number that meant nothing.
• The frontier model's best configuration is the one reported. Worked examples were tried and made it worse — 0.8150 against 0.8296 — because the list of types is already in the prompt, so the examples add nothing and pull answers toward the few types shown. The stronger of the two is what it is compared against.

The most useful output is not the accuracy

70 of the 77 request types improved and 7 went backwards, all of them on types with very few test examples, where a single message moves the score. But the finding worth paying for is which messages both models get wrong, because they are consistently the same ones:

Read those pairs as a customer would. "Where is my card" and "when will my card arrive" are not obviously different requests, and neither are the two top-up cases. These are not model defects; they are places where the categories themselves overlap. A team routing tickets by hand makes the same call inconsistently, and that inconsistency is in the training data because it is in the business.

That is what the audit is for. It returns the list of distinctions your own taxonomy cannot support — a finding about your operation, not about the model — and merging two categories that were never really separate usually buys more accuracy than any amount of further training.

How this is deployed

These accuracy rates support a review-assisted workflow: the model fills ten fields, a person checks and corrects rather than keys from scratch, and the fields that need the most attention are known in advance rather than discovered later.

It is not unattended automation, and no model tested here is — including Claude Opus 5, at any price, with or without examples. Across all three, no contract came back with all ten fields simultaneously correct. That is the honest ceiling of the task today, and knowing it before you commit is the difference between a project that lands and one that quietly disappoints. If someone shows you a single accuracy figure without telling you this, ask them for it.

Cost

API figures are token counts from the runs against the provider's published prices. The serving figure is measured throughput against a rented GPU's hourly rate.

The comparison a buyer should actually make

The figures above are the providers' standard rates, which is what a task costs before anyone tunes it. A team that knows what it is doing would not run it that way, so here is the same comparison with the frontier bill optimised as far as it goes. Prompt caching charges a repeated instruction once instead of on every call; it was measured, not estimated.

Every figure in that table was measured, including the ones that disappoint. Caching cut the routing bill by 74%: the same list of 77 request types goes out with every message, so 97% of each call is identical and cacheable.

Cold extraction cannot use caching at all, and the reason is a detail worth knowing before you budget around it. Anthropic will only cache a shared prefix of at least 512 tokens on this model. A contract prompt repeats just the instruction and the field list — around 350 tokens — so it never qualifies, and a cached run of it came back costing marginally more, not less. Add three worked examples and the shared block clears the threshold easily: the same task then falls from $63.18 to $28.27, a 55% saving.

The rule that generalises: what caches is the part of your prompt that never changes, and it has to be substantial before the provider will hold it. If your calls are mostly a fixed instruction, your real bill is well below the headline and the audit prices against that. If they are mostly your own documents, caching will not rescue them.

Two more things belong here rather than in a footnote. Batch processing takes another 50% off both figures if your work tolerates up to a day's delay — which routing does not, since the point of it is to move a message now. And the same optimisation applies on our side: the figures above use a pay-per-second on-demand card, where a dedicated one of the same class costs roughly a quarter as much.

The gap narrows and does not close. Against the cheapest frontier bill in that table, a model you own is about nine times cheaper on routing and about twenty-two times cheaper on extraction — caching helps the frontier most exactly where its bill was smallest to begin with. Batch processing would halve both sides of that comparison again, and is left out of these multiples rather than folded quietly into one of them.

On routing, though, cost is the smaller argument. The reason to own that model is that it makes 54% fewer mistakes and never returns anything unparseable.

It is also deliberately the most expensive of four measurements of the same setup, and it uses a pay-per-second provider's on-demand price — a dedicated rented card of the same class costs roughly a quarter as much. The saving is quoted conservatively because the figure has to survive contact with your invoice, not just with this page.

Your own numbers will differ, because they depend on your documents and your volume. Measuring them on your data, before you commit to anything, is what the audit is.