Skip to main content
Most teams pick one big model on day one and pay big-model prices for small-model work. Twelve of your real prompts are enough to see the split: on ours, a 24B model and a free-tier model covered all four kinds between them. This recipe runs every prompt on five models at once, grades what can be graded exactly and has a judge score the rest, then writes a routing table that sends each kind to the cheapest model within 5% of the best. The whole arena cost under a cent.

Build it in your project

Paste this line into your agent (Claude Code, Codex, Cursor, OpenCode, Cline). It sets up your key, then builds the pipeline in your project:

The code

One file per language. Both read RUNCRATE_API_KEY from the environment; nothing secret is in the code.

What the proof run cost

The Python file on twelve prompts in four kinds (extract, classify, math, write). Measured on 2026-10-31 against api.runcrate.ai; costs are what the account was billed, read from each response’s X-LLM-Cost header or usage.cost. Routes on the proof run: classify to gpt-oss-120b, extract to Mistral Small 3.2, math to gpt-oss-120b, write to Mistral Small 3.2. The TypeScript run picked the same four, in 18.1 s for $0.0063. Scores per kind and the list price per 1,000 answers are in leaderboard.json.

Scale it

  • Re-run it weekly from a schedule: prices and models change, and so does the routing table.
  • More prompts per kind give a steadier table; a few dozen is plenty.
  • The judge is the expensive part: judge a sample, not every answer.

Gotchas found in the proof run

  • The three free-tier contenders were billed on the proof run as paid requests: DeepSeek V4.1 Flash and Qwen3.8 Flash answers at half the list price, gpt-oss-120b at the full list price. The table prices every answer at the list price so the comparison is fair.
  • /v1/models and the catalog (/api/models/catalog) both list the billed prices: ×1.05 on text, vision, embeddings, speech and transcription; ×1.25 on images and video. The script uses the catalog.
  • Grade exact answers loosely: models wrap them in backticks, add a dollar sign or a period.