Method
The eval harness is deliberately strict. It scores every field of every document for precision, recall and F1. It also reports two numbers most benchmarks leave out: how often the model gets an entire record right, and how often it returns output that doesn't parse at all.
The same scoring runs against the frontier baseline and the fine-tune. A wrong value is penalised twice, once as a false positive and once as a false negative, so a model can't lift its score by guessing. Agreement on empty fields is tracked separately and never inflates a score.
What gets measured on your task
The same scoring is applied to your endpoint, on documents held back from training so nothing has seen them before. You are shown four things per field, not one headline:
- Field-level accuracy — precision and recall for every field separately, so you can see which parts of the task are solved and which are not.
- Whole-record accuracy — how often all fields are simultaneously right. This is the number that decides whether a workflow can run unattended, and it is usually far lower than the headline.
- Malformed-output rate — how often the model returns something that does not parse at all.
- Cost per 1,000 documents, measured on both sides rather than estimated.
How that scoring performed against a frontier model on two public datasets — one of contracts, one of customer messages — including the places the frontier model still wins: the benchmark in full.
What providers have retired, repriced and announced
The dates below are the providers' own, taken from their published deprecation and pricing pages; events still ahead are marked as announced. The list is not an argument that renting always costs more; provider prices move down as well as up. What it shows is that the terms move on the provider's schedule, not yours.
| Date | Event |
|---|---|
| 5 Jan 2026 | Anthropic retired Claude Opus 3. |
| 7 May 2026 | OpenAI stopped organisations that had never fine-tuned from creating new fine-tuning jobs. |
| 15 Jun 2026 | Anthropic retired Claude Sonnet 4 and Claude Opus 4, notified on 14 April — roughly two months' notice. |
| 2 Jul 2026 | OpenAI extended that restriction to organisations with no fine-tuned-model inference in the previous 60 days. |
| 23 Jul 2026 | OpenAI shut down 15 model snapshots in one wave, including the gpt-5.1 variants and gpt-5-codex, announced on 22 April — roughly three months' notice. |
| 1 Sep 2026 | Announced: Claude Sonnet 5's introductory price of $2/$10 per million tokens ends on 31 August; the standard $3/$15 applies from 1 September — a 50% rise on the same model. |
| 23 Oct 2026 | Announced: OpenAI's fine-tuned GPT-3.5 Turbo and GPT-4 models shut down with their base models. Fine-tunes built on those base models stop running on that date. |
| 11 Dec 2026 | Announced: OpenAI retires its GPT-5 and o3 snapshots, notified on 11 June — six months' notice. |
| 6 Jan 2027 | Announced: OpenAI's active existing fine-tuning customers can no longer create new fine-tuning jobs — the last of three restrictions, after those of 7 May and 2 July 2026. |
A model whose weights you own cannot be retired or repriced by anyone but you. That is Coldstill's conclusion from these published schedules, not a position any provider has taken.
Every date and price above comes from the providers' own pages: OpenAI's model deprecations page, Anthropic's model deprecations page and Anthropic's pricing page, all checked 2026-08-04. Provider documentation gets rewritten as schedules move, so each is also fixed as an archived copy: OpenAI, Anthropic's deprecations and Anthropic's pricing as archived — so a date on this page can be checked against what the provider published at the time, not only against what they publish now.
What an engagement verifies
Every migration ships the harness itself, and quality is the acceptance condition, not a hope: nothing switches until the comparison shows parity on your data. Self-run deployments re-run the comparison on your own hardware to reproduce the delivered numbers; hosted deployments re-run it before switchover and weekly thereafter, with the results reported to you. If a deployment is misconfigured, it shows up as a score drop.
Get a free read on your task
This is the scoring the free read applies to your task, on your numbers.
The enquiry form takes six questions and a written read comes back within a working day. Email works just as well: [email protected].