Receipts and invoices to JSON to a webhook

Expense tools, bookkeeping and procurement all start with a photo of a receipt. A vision model with a strict JSON schema reads it, but the check that matters is arithmetic: do the lines add up to the subtotal, and the subtotal plus tax to the total? This pipeline runs the cheap model first, only escalates to the large one when the numbers disagree, and posts each result to your endpoint signed, keyed by the image’s hash so a retry never books it twice. Three receipts cost six hundredths of a cent.

Build it in your project

Paste this line into your agent (Claude Code, Codex, Cursor, OpenCode, Cline). It sets up your key, then builds the pipeline in your project:

Add Runcrate's Qwen3 VL 30B A3B to this project: run curl -fsSL "https://runcrate.ai/agents.md?model=Qwen%2FQwen3-VL-30B-A3B-Instruct&intent=build" and follow it. Then build: a script that extracts receipts and invoices in a folder to strict JSON (vendor, date, currency, lines, subtotal, tax, total) with Qwen3-VL 30B, checks that the lines add up, retries once on Qwen3-VL 235B when they do not, and posts each result to WEBHOOK_URL with an HMAC signature and an idempotency key.

The code

One file per language. Both read RUNCRATE_API_KEY from the environment; nothing secret is in the code.

"""Receipts and invoices to checked JSON, posted to your webhook.

pip install openai requests
RUNCRATE_API_KEY=... WEBHOOK_URL=https://your.app/hooks/receipts WEBHOOK_SECRET=... python receipts_to_webhook.py ./receipts
"""
import base64, hashlib, hmac, json, mimetypes, os, sys, time
from pathlib import Path

import requests
from openai import OpenAI

client = OpenAI(base_url="https://api.runcrate.ai/v1", api_key=os.environ["RUNCRATE_API_KEY"], max_retries=4)
FAST = "Qwen/Qwen3-VL-30B-A3B-Instruct"     # first pass
STRONG = "Qwen/Qwen3-VL-235B-A22B-Instruct"  # only when the numbers don't add up

SCHEMA = {
    "type": "object",
    "additionalProperties": False,
    "required": ["vendor", "date", "currency", "lines", "subtotal", "tax", "total"],
    "properties": {
        "vendor": {"type": "string"},
        "date": {"type": "string", "description": "ISO 8601, YYYY-MM-DD"},
        "currency": {"type": "string", "description": "ISO 4217 code"},
        "lines": {
            "type": "array",
            "items": {
                "type": "object",
                "additionalProperties": False,
                "required": ["description", "quantity", "amount"],
                "properties": {"description": {"type": "string"}, "quantity": {"type": "number"}, "amount": {"type": "number"}},
            },
        },
        "subtotal": {"type": "number"},
        "tax": {"type": "number"},
        "total": {"type": "number"},
    },
}

def extract(path: Path, model: str) -> tuple[dict, float]:
    mime = mimetypes.guess_type(path.name)[0] or "image/jpeg"
    data_url = f"data:{mime};base64,{base64.b64encode(path.read_bytes()).decode()}"
    raw = client.chat.completions.with_raw_response.create(
        model=model,
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Extract this receipt. Amounts are line totals as printed. Use the currency the receipt shows."},
                {"type": "image_url", "image_url": {"url": data_url}},
            ],
        }],
        response_format={"type": "json_schema", "json_schema": {"name": "receipt", "strict": True, "schema": SCHEMA}},
        temperature=0,
    )
    reply = raw.parse()
    return json.loads(reply.choices[0].message.content), float(raw.headers.get("x-llm-cost", 0))

def adds_up(r: dict) -> bool:
    lines = round(sum(l["amount"] for l in r["lines"]), 2)
    return abs(lines - r["subtotal"]) < 0.02 and abs(r["subtotal"] + r["tax"] - r["total"]) < 0.02

def post_webhook(payload: dict) -> int:
    url = os.environ.get("WEBHOOK_URL")
    if not url:
        return 0
    body = json.dumps(payload, separators=(",", ":")).encode()
    sig = hmac.new(os.environ.get("WEBHOOK_SECRET", "").encode(), body, hashlib.sha256).hexdigest()
    headers = {"Content-Type": "application/json", "X-Signature-SHA256": sig, "Idempotency-Key": payload["sha256"]}
    for attempt in range(5):
        try:
            res = requests.post(url, data=body, headers=headers, timeout=10)
            if res.status_code < 500 and res.status_code != 429:
                return res.status_code
        except requests.RequestException:
            pass
        time.sleep(2 ** attempt)
    raise RuntimeError(f"webhook {url} kept failing")

def main(folder: str) -> None:
    ledger = []
    for path in sorted(p for p in Path(folder).iterdir() if p.suffix.lower() in {".jpg", ".jpeg", ".png", ".webp"}):
        t0 = time.time()
        receipt, cost = extract(path, FAST)
        model = FAST
        if not adds_up(receipt):  # escalate once, to the bigger model
            receipt, extra = extract(path, STRONG)
            cost += extra
            model = STRONG
        payload = {
            "file": path.name,
            "sha256": hashlib.sha256(path.read_bytes()).hexdigest(),
            "model": model,
            "needs_review": not adds_up(receipt),
            "receipt": receipt,
        }
        status = post_webhook(payload)
        secs = time.time() - t0
        ledger.append({"file": path.name, "model": model, "seconds": round(secs, 2), "cost_usd": cost, "webhook": status, "total": receipt["total"]})
        print(f"{path.name}: {receipt['vendor']} {receipt['total']} {receipt['currency']}  {model}  {secs:.1f}s  ${cost:.6f}  webhook {status}")
    Path("run.json").write_text(json.dumps(ledger, indent=2))
    print(f"total ${sum(r['cost_usd'] for r in ledger):.6f} for {len(ledger)} receipts")

if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "receipts")

// Receipts and invoices to checked JSON, posted to your webhook.
//
// npm install openai
// RUNCRATE_API_KEY=... WEBHOOK_URL=https://your.app/hooks/receipts WEBHOOK_SECRET=... npx tsx receipts-to-webhook.ts ./receipts
import { createHash, createHmac } from "node:crypto";
import { readdirSync, readFileSync, writeFileSync } from "node:fs";
import { extname, join } from "node:path";
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://api.runcrate.ai/v1", apiKey: process.env.RUNCRATE_API_KEY, maxRetries: 4 });
const FAST = "Qwen/Qwen3-VL-30B-A3B-Instruct"; // first pass
const STRONG = "Qwen/Qwen3-VL-235B-A22B-Instruct"; // only when the numbers don't add up

type Receipt = {
  vendor: string;
  date: string;
  currency: string;
  lines: { description: string; quantity: number; amount: number }[];
  subtotal: number;
  tax: number;
  total: number;
};

const SCHEMA = {
  type: "object",
  additionalProperties: false,
  required: ["vendor", "date", "currency", "lines", "subtotal", "tax", "total"],
  properties: {
    vendor: { type: "string" },
    date: { type: "string", description: "ISO 8601, YYYY-MM-DD" },
    currency: { type: "string", description: "ISO 4217 code" },
    lines: {
      type: "array",
      items: {
        type: "object",
        additionalProperties: false,
        required: ["description", "quantity", "amount"],
        properties: { description: { type: "string" }, quantity: { type: "number" }, amount: { type: "number" } },
      },
    },
    subtotal: { type: "number" },
    tax: { type: "number" },
    total: { type: "number" },
  },
};

const MIME: Record<string, string> = { ".jpg": "image/jpeg", ".jpeg": "image/jpeg", ".png": "image/png", ".webp": "image/webp" };

async function extract(path: string, model: string): Promise<[Receipt, number]> {
  const dataUrl = `data:${MIME[extname(path).toLowerCase()]};base64,${readFileSync(path).toString("base64")}`;
  const { data, response } = await client.chat.completions
    .create({
      model,
      messages: [
        {
          role: "user",
          content: [
            { type: "text", text: "Extract this receipt. Amounts are line totals as printed. Use the currency the receipt shows." },
            { type: "image_url", image_url: { url: dataUrl } },
          ],
        },
      ],
      response_format: { type: "json_schema", json_schema: { name: "receipt", strict: true, schema: SCHEMA } },
      temperature: 0,
    })
    .withResponse();
  return [JSON.parse(data.choices[0].message.content ?? "{}"), Number(response.headers.get("x-llm-cost") ?? 0)];
}

function addsUp(r: Receipt): boolean {
  const lines = Math.round(r.lines.reduce((s, l) => s + l.amount, 0) * 100) / 100;
  return Math.abs(lines - r.subtotal) < 0.02 && Math.abs(r.subtotal + r.tax - r.total) < 0.02;
}

async function postWebhook(payload: { sha256: string }): Promise<number> {
  const url = process.env.WEBHOOK_URL;
  if (!url) return 0;
  const body = JSON.stringify(payload);
  const sig = createHmac("sha256", process.env.WEBHOOK_SECRET ?? "").update(body).digest("hex");
  for (let attempt = 0; attempt < 5; attempt++) {
    try {
      const res = await fetch(url, {
        method: "POST",
        body,
        headers: { "Content-Type": "application/json", "X-Signature-SHA256": sig, "Idempotency-Key": payload.sha256 },
        signal: AbortSignal.timeout(10_000),
      });
      if (res.status < 500 && res.status !== 429) return res.status;
    } catch {
      // network error: retry
    }
    await new Promise((r) => setTimeout(r, 2 ** attempt * 1000));
  }
  throw new Error(`webhook ${url} kept failing`);
}

const folder = process.argv[2] ?? "receipts";
const ledger: { file: string; model: string; seconds: number; cost_usd: number; webhook: number; total: number }[] = [];
for (const name of readdirSync(folder).filter((f) => MIME[extname(f).toLowerCase()]).sort()) {
  const path = join(folder, name);
  const t0 = Date.now();
  let [receipt, cost] = await extract(path, FAST);
  let model = FAST;
  if (!addsUp(receipt)) {
    // escalate once, to the bigger model
    const [again, extra] = await extract(path, STRONG);
    receipt = again;
    cost += extra;
    model = STRONG;
  }
  const sha256 = createHash("sha256").update(readFileSync(path)).digest("hex");
  const status = await postWebhook({ file: name, sha256, model, needs_review: !addsUp(receipt), receipt } as { sha256: string });
  const seconds = (Date.now() - t0) / 1000;
  ledger.push({ file: name, model, seconds, cost_usd: cost, webhook: status, total: receipt.total });
  console.log(`${name}: ${receipt.vendor} ${receipt.total} ${receipt.currency}  ${model}  ${seconds.toFixed(1)}s  $${cost.toFixed(6)}  webhook ${status}`);
}
writeFileSync("run.json", JSON.stringify(ledger, null, 2));
console.log(`total $${ledger.reduce((s, r) => s + r.cost_usd, 0).toFixed(6)} for ${ledger.length} receipts`);

What the proof run cost

The Python file on three receipt photos made for the test (rotated and slightly blurred), with known totals. Measured on 2026-10-23 against api.runcrate.ai; costs are what the account was billed, read from each response’s X-LLM-Cost header or usage.cost.

Step	Model or tool	Time	Cost
Blue Door Coffee, GBP 10.05	Qwen/Qwen3-VL-30B-A3B-Instruct	5.9 s	$0.000211
Hardware Hub, USD 48.97	Qwen/Qwen3-VL-30B-A3B-Instruct	5.2 s	$0.000213
Taqueria La Luna, USD 24.24	Qwen/Qwen3-VL-30B-A3B-Instruct	5.2 s	$0.000205
Total, 3 receipts, 3 webhooks (signature verified)		16 s	$0.000629

All three totals, vendors, dates and currencies were exact, in both languages (TypeScript: $0.000629, 17 s). Every receipt added up, so the escalation to the 235B model did not run.

Scale it

Thousands a day: each receipt is independent; run a thread pool, or a job group over a folder on a volume.
PDF invoices: render each page to an image first (pdftoppm -r 150), then send the pages in one message.
Cap the spend with a key limit; a broken loop stops at your number.

Gotchas found in the proof run

inclusionAI/Ling-3.0-flash-VL is cheaper but ignored response_format and answered in prose; Qwen3-VL honoured the strict schema every time.
Ask for line totals “as printed”: otherwise the model sometimes multiplies quantity by unit price itself and disagrees with the receipt.
Sign the exact bytes you send (separators=(",", ":") in Python) or the receiver’s HMAC will not match.

Start here

Recipes: Media

Recipes: Voice

Recipes: Documents and RAG

Recipes: Agents

Recipes: Training and serving

Text and agents

Vision and documents

Images

Video

Speech

GPUs

Storage

Receipts and invoices to JSON to a webhook

Build it in your project

The code

What the proof run cost

Scale it

Gotchas found in the proof run

​Build it in your project

​The code

​What the proof run cost

​Scale it

​Gotchas found in the proof run

Build it in your project

The code

What the proof run cost

Scale it

Gotchas found in the proof run