> ## Documentation Index
> Fetch the complete documentation index at: https://runcrate.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Synthetic data to a LoRA, served by vLLM, scored against the base
> A large model writes training data, a 0.5B model learns it with LoRA on a GPU job, vLLM serves the base and the adapter, and a judge scores them.
A small model that does one job well is cheaper and faster than a big one that does everything. The hard parts are the data and the comparison. Here a large model on the free tier writes the training pairs, a 0.5B model learns them with LoRA on a rented GPU, and the same job serves the base model and the adapter side by side from one vLLM server, so the comparison is fair. A judge on the inference API then scores both on prompts neither saw in training.
```mermaid theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
flowchart LR
T[Teacher
gpt-oss-120b, free tier] --> D[train.jsonl / eval.jsonl]
D --> U[upload code + data]
U --> J[GPU job]
subgraph J[GPU job, 10 min cap]
L[LoRA on Qwen2.5 0.5B] --> V[vLLM: base + adapter]
V --> P[predictions]
end
P --> A[artifacts]
A --> S[judge
gpt-oss-120b]
S --> R[base vs LoRA]
```
## Build it in your project
Paste this line into your agent (Claude Code, Codex, Cursor, OpenCode, Cline). It sets up your key, then builds the pipeline in your project:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
Add Runcrate's gpt-oss 120B to this project: run curl -fsSL "https://runcrate.ai/agents.md?model=openai%2Fgpt-oss-120b&intent=build" and follow it. Then build: a pipeline that has gpt-oss-120b write request-to-command pairs for a CLI, uploads them with a training script to a Runcrate GPU job (ubuntu-inference template, 10-minute cap), trains a LoRA on Qwen2.5-0.5B-Instruct, serves base and adapter from one vLLM server in the job, predicts the held-out set, then scores both with a judge through the inference API.
```
## The code
One file per language. Both read `RUNCRATE_API_KEY` from the environment; nothing secret is in the code.
```python lora_finetune_serve_eval.py theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
"""Synthetic data -> a LoRA fine-tune on a GPU job -> vLLM serving the base and the adapter -> scored against each other.
The task: turn a plain-English request into the exact `runcrate` CLI command. A large model writes the training
data, a 0.5B model learns it on a rented GPU, and a judge on the inference API scores base against fine-tune.
pip install openai runcrate-sdk
RUNCRATE_API_KEY=... python lora_finetune_serve_eval.py
The same file runs on the GPU as `python lora_finetune_serve_eval.py --job` (the driver uploads it).
"""
import json, os, random, re, shutil, subprocess, sys, tempfile, time
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
BASE = "Qwen/Qwen2.5-0.5B-Instruct" # Hugging Face id: this model runs on the job's GPU, not the API
TEACHER = "openai/gpt-oss-120b" # writes the data, on the free tier
JUDGE = "openai/gpt-oss-120b"
GPUS = ["A6000", "L40S", "L40", "A4000", "RTX4000Ada"] # tried in order
SYSTEM = "Reply with exactly one runcrate CLI command and nothing else."
CHEATSHEET = """runcrate instances create --name NAME --gpu GPU [--gpu-count N] [--template ubuntu-devbox|ubuntu-train|ubuntu-inference] [--max-runtime 4h] [--idle-stop 30m]
runcrate instances list | runcrate instances delete NAME | runcrate instances types --gpu GPU
runcrate ssh NAME -- COMMAND | runcrate cp ./local NAME:/remote | runcrate cp NAME:/remote ./local
runcrate run --gpu GPU [--gpu-count N] [--template T] [--volume ID] [--max-runtime 2h] [--retries N] [--env K=V] [--detach] -- COMMAND
runcrate jobs list [--status running] | runcrate jobs logs ID -f | runcrate jobs cancel ID | runcrate jobs download ID
runcrate schedule create --cron 'M H * * *' --tz ZONE --name NAME --gpu GPU --max-runtime 2h [--volume ID] -- COMMAND
runcrate schedule list | runcrate schedule delete ID
runcrate volumes create --name NAME --size GB --region REGION | runcrate volumes list | runcrate volumes delete ID
runcrate billing balance | runcrate billing usage | runcrate secrets list | runcrate mcp install --client cursor
GPUs: H100, H200, B200, A100, A100_80G, L40S, L40, A6000, A4000, RTX4000Ada. Times like 90m, 2h, 1h30m."""
# ─────────────────────────────── on the GPU: train, serve, predict ───────────────────────────────
def job() -> None:
import threading, urllib.request
import torch
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer
out = Path(os.environ.get("RUNCRATE_OUTPUT_DIR", "out")); out.mkdir(parents=True, exist_ok=True)
train = [json.loads(l) for l in open("train.jsonl")]
evals = [json.loads(l) for l in open("eval.jsonl")]
marks = {"gpu": torch.cuda.get_device_name(0)}
print(f"{marks['gpu']}: {len(train)} train, {len(evals)} eval", flush=True)
t0 = time.time()
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16).cuda()
model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]))
rows = []
for ex in train: # loss on the answer only
prompt = tok.apply_chat_template([{"role": "system", "content": SYSTEM}, {"role": "user", "content": ex["request"]}], tokenize=False, add_generation_prompt=True)
p_ids = tok(prompt, add_special_tokens=False).input_ids
a_ids = tok(ex["command"] + tok.eos_token, add_special_tokens=False).input_ids
rows.append((p_ids + a_ids, [-100] * len(p_ids) + a_ids))
opt = torch.optim.AdamW(model.parameters(), lr=2e-4)
model.train()
for epoch in range(3):
random.Random(epoch).shuffle(rows)
for i in range(0, len(rows), 8):
batch = rows[i:i + 8]
n = max(len(r[0]) for r in batch)
ids = torch.tensor([r[0] + [tok.pad_token_id] * (n - len(r[0])) for r in batch]).cuda()
labels = torch.tensor([r[1] + [-100] * (n - len(r[1])) for r in batch]).cuda()
loss = model(input_ids=ids, attention_mask=(ids != tok.pad_token_id).long(), labels=labels).loss
loss.backward(); opt.step(); opt.zero_grad()
print(f"epoch {epoch + 1} loss {loss.item():.4f}", flush=True)
model.save_pretrained(out / "adapter")
marks["train_seconds"] = round(time.time() - t0, 1)
del model, opt; torch.cuda.empty_cache()
# Serve the base and the adapter from one vLLM server, as you would on an instance.
t0 = time.time()
server = subprocess.Popen([sys.executable, "-m", "vllm.entrypoints.openai.api_server", "--model", BASE, "--port", "8000",
"--enable-lora", "--lora-modules", f"cli={out / 'adapter'}", "--max-lora-rank", "16",
"--max-model-len", "1024", "--gpu-memory-utilization", "0.5", "--enforce-eager"],
stdout=open(out / "vllm.log", "w"), stderr=subprocess.STDOUT)
for _ in range(300):
try:
urllib.request.urlopen("http://127.0.0.1:8000/health", timeout=2); break
except Exception:
if server.poll() is not None:
raise RuntimeError("vLLM exited: see vllm.log")
time.sleep(1)
marks["vllm_ready_seconds"] = round(time.time() - t0, 1)
def ask(model_name: str, request: str) -> str:
body = json.dumps({"model": model_name, "temperature": 0, "max_tokens": 96,
"messages": [{"role": "system", "content": SYSTEM}, {"role": "user", "content": request}]}).encode()
req = urllib.request.Request("http://127.0.0.1:8000/v1/chat/completions", data=body, headers={"Content-Type": "application/json"})
return json.loads(urllib.request.urlopen(req, timeout=60).read())["choices"][0]["message"]["content"].strip()
t0 = time.time()
preds = []
with ThreadPoolExecutor(max_workers=16) as pool:
for name in (BASE, "cli"):
for ex, got in zip(evals, pool.map(lambda e: ask(name, e["request"]), evals)):
preds.append({"model": "base" if name == BASE else "lora", **ex, "prediction": got})
marks["predict_seconds"] = round(time.time() - t0, 1)
server.terminate()
(out / "predictions.jsonl").write_text("".join(json.dumps(p) + "\n" for p in preds))
(out / "marks.json").write_text(json.dumps(marks))
print(json.dumps(marks), flush=True)
# ─────────────────────────────── on your machine: data, job, score ───────────────────────────────
def driver() -> None:
from openai import OpenAI
from runcrate import Runcrate
api = OpenAI(base_url="https://api.runcrate.ai/v1", api_key=os.environ["RUNCRATE_API_KEY"], max_retries=4)
rc = Runcrate(api_key=os.environ["RUNCRATE_API_KEY"])
ledger = {}
# 1. Synthetic data: varied requests, one correct command each.
t0, cost = time.time(), 0.0
schema = {"type": "object", "additionalProperties": False, "required": ["pairs"], "properties": {"pairs": {"type": "array", "items": {
"type": "object", "additionalProperties": False, "required": ["request", "command"],
"properties": {"request": {"type": "string"}, "command": {"type": "string"}}}}}}
def batch(seed: int) -> tuple[list, float]:
r = api.chat.completions.create(model=TEACHER, temperature=1.0, seed=seed, response_format={"type": "json_schema", "json_schema": {"name": "pairs", "strict": True, "schema": schema}},
messages=[{"role": "user", "content": f"CLI reference:\n{CHEATSHEET}\n\nWrite 20 different requests a developer might type, casual or terse, "
f"each with the single correct command. Vary GPUs, names, times, flags. Batch {seed}."}])
return json.loads(r.choices[0].message.content)["pairs"], float(getattr(r.usage, "cost", 0) or 0)
with ThreadPoolExecutor(max_workers=6) as pool:
results = list(pool.map(batch, range(12)))
seen, pairs = set(), []
for got, c in results:
cost += c
for p in got:
if p["command"].startswith("runcrate ") and p["request"].lower() not in seen:
seen.add(p["request"].lower()); pairs.append(p)
random.Random(0).shuffle(pairs)
evals, train = pairs[:40], pairs[40:]
ledger["data"] = {"seconds": round(time.time() - t0, 1), "cost_usd": cost, "train": len(train), "eval": len(evals)}
print(f"data: {len(train)} train, {len(evals)} eval, {ledger['data']['seconds']}s, ${cost:.6f}")
# 2. The job: this file and the data, on the first GPU that is free.
code = Path(tempfile.mkdtemp())
shutil.copy(__file__, code / "recipe.py")
(code / "train.jsonl").write_text("".join(json.dumps(p) + "\n" for p in train))
(code / "eval.jsonl").write_text("".join(json.dumps(p) + "\n" for p in evals))
upload = rc.jobs.uploads.upload(str(code))
t0 = time.time()
submitted = rc.jobs.run(command="pip install -q peft && python3 recipe.py --job", gpu_type=GPUS, template="ubuntu-inference",
max_runtime_minutes=10, max_retries=1, code={"upload_id": upload.upload_id}, name="lora-cli-recipe",
tags={"recipe": "lora-finetune-serve-eval"})
print(f"job {submitted.id} submitted")
for chunk in rc.jobs.follow_logs(submitted.id):
print(chunk.text, end="", flush=True)
done = rc.jobs.wait(submitted.id, timeout=1800)
job_cost = sum(a.cost_usd or 0 for a in done.attempts)
ledger["job"] = {"id": done.id, "status": done.status, "gpu": done.gpu_type, "seconds": round(time.time() - t0, 1), "cost_usd": job_cost}
print(f"\njob {done.status} on {done.gpu_type}, {ledger['job']['seconds']}s from submit, ${job_cost:.4f}")
if done.status != "succeeded":
raise SystemExit(f"job ended {done.status}: {done.error}")
out = Path("out"); out.mkdir(exist_ok=True)
for a in rc.jobs.artifacts.list(done.id):
dest = out / a.path; dest.parent.mkdir(parents=True, exist_ok=True)
rc.jobs.artifacts.download(done.id, a.path, to=dest)
preds = [json.loads(l) for l in (out / "predictions.jsonl").read_text().splitlines()]
ledger["job"] |= json.loads((out / "marks.json").read_text())
# 3. Score: exact match, then a judge for commands that differ only in form.
t0, cost = time.time(), 0.0
norm = lambda s: re.sub(r"\s+", " ", s.strip().strip("`").replace('"', "'"))
def judge(p: dict) -> tuple[bool, float]:
if norm(p["prediction"]) == norm(p["command"]):
return True, 0.0
r = api.chat.completions.create(model=JUDGE, temperature=0, response_format={"type": "json_schema", "json_schema": {"name": "v", "strict": True, "schema": {
"type": "object", "additionalProperties": False, "required": ["same"], "properties": {"same": {"type": "boolean"}}}}},
messages=[{"role": "user", "content": f"CLI reference:\n{CHEATSHEET}\n\nRequest: {p['request']}\nReference: {p['command']}\nCandidate: {p['prediction']}\n"
"Does the candidate do exactly what the request asks, as the reference does? Flag order does not matter."}])
return json.loads(r.choices[0].message.content)["same"], float(getattr(r.usage, "cost", 0) or 0)
with ThreadPoolExecutor(max_workers=8) as pool:
verdicts = list(pool.map(judge, preds))
score = {}
for p, (ok, c) in zip(preds, verdicts):
cost += c; p["correct"] = ok
s = score.setdefault(p["model"], {"correct": 0, "exact": 0, "n": 0})
s["n"] += 1; s["correct"] += ok; s["exact"] += norm(p["prediction"]) == norm(p["command"])
ledger["score"] = {"seconds": round(time.time() - t0, 1), "cost_usd": cost, **score}
for m, s in score.items():
print(f"{m:<5} {s['correct']}/{s['n']} correct ({s['exact']} exact)")
ledger["total_usd"] = ledger["data"]["cost_usd"] + job_cost + cost
Path("run.json").write_text(json.dumps(ledger, indent=2))
(out / "scored.jsonl").write_text("".join(json.dumps(p) + "\n" for p in preds))
print(f"total ${ledger['total_usd']:.4f}")
if __name__ == "__main__":
job() if "--job" in sys.argv else driver()
```
```typescript lora-finetune-serve-eval.ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
// Synthetic data -> a LoRA fine-tune on a GPU job -> vLLM serving the base and the adapter -> scored against each other.
//
// The task: turn a plain-English request into the exact `runcrate` CLI command. A large model writes the training
// data, a 0.5B model learns it on a rented GPU, and a judge on the inference API scores base against fine-tune.
//
// npm install openai @runcrate/sdk (and tar on PATH)
// RUNCRATE_API_KEY=... npx tsx lora-finetune-serve-eval.ts [--dry-run]
// The GPU side is Python (PyTorch, PEFT, vLLM): this file writes it as job.py and uploads it with the data.
import { execFileSync } from "node:child_process";
import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import OpenAI from "openai";
import Runcrate from "@runcrate/sdk";
const TEACHER = "openai/gpt-oss-120b"; // writes the data, on the free tier
const JUDGE = "openai/gpt-oss-120b";
const GPUS = ["A6000", "L40S", "L40", "A4000", "RTX4000Ada"]; // tried in order
const CHEATSHEET = "runcrate instances create --name NAME --gpu GPU [--gpu-count N] [--template ubuntu-devbox|ubuntu-train|ubuntu-inference] [--max-runtime 4h] [--idle-stop 30m]\nruncrate instances list | runcrate instances delete NAME | runcrate instances types --gpu GPU\nruncrate ssh NAME -- COMMAND | runcrate cp ./local NAME:/remote | runcrate cp NAME:/remote ./local\nruncrate run --gpu GPU [--gpu-count N] [--template T] [--volume ID] [--max-runtime 2h] [--retries N] [--env K=V] [--detach] -- COMMAND\nruncrate jobs list [--status running] | runcrate jobs logs ID -f | runcrate jobs cancel ID | runcrate jobs download ID\nruncrate schedule create --cron 'M H * * *' --tz ZONE --name NAME --gpu GPU --max-runtime 2h [--volume ID] -- COMMAND\nruncrate schedule list | runcrate schedule delete ID\nruncrate volumes create --name NAME --size GB --region REGION | runcrate volumes list | runcrate volumes delete ID\nruncrate billing balance | runcrate billing usage | runcrate secrets list | runcrate mcp install --client cursor\nGPUs: H100, H200, B200, A100, A100_80G, L40S, L40, A6000, A4000, RTX4000Ada. Times like 90m, 2h, 1h30m.";
const JOB_PY = "\"\"\"The GPU side of the LoRA recipe: train, serve base and adapter with vLLM, predict. Run as: python3 recipe.py --job\"\"\"\nimport json, os, random, re, shutil, subprocess, sys, tempfile, time\nfrom concurrent.futures import ThreadPoolExecutor\nfrom pathlib import Path\n\nBASE = \"Qwen/Qwen2.5-0.5B-Instruct\" # Hugging Face id: this model runs on the job's GPU, not the API\nTEACHER = \"openai/gpt-oss-120b\" # writes the data, on the free tier\nJUDGE = \"openai/gpt-oss-120b\"\nGPUS = [\"A6000\", \"L40S\", \"L40\", \"A4000\", \"RTX4000Ada\"] # tried in order\nSYSTEM = \"Reply with exactly one runcrate CLI command and nothing else.\"\n\nCHEATSHEET = \"\"\"runcrate instances create --name NAME --gpu GPU [--gpu-count N] [--template ubuntu-devbox|ubuntu-train|ubuntu-inference] [--max-runtime 4h] [--idle-stop 30m]\nruncrate instances list | runcrate instances delete NAME | runcrate instances types --gpu GPU\nruncrate ssh NAME -- COMMAND | runcrate cp ./local NAME:/remote | runcrate cp NAME:/remote ./local\nruncrate run --gpu GPU [--gpu-count N] [--template T] [--volume ID] [--max-runtime 2h] [--retries N] [--env K=V] [--detach] -- COMMAND\nruncrate jobs list [--status running] | runcrate jobs logs ID -f | runcrate jobs cancel ID | runcrate jobs download ID\nruncrate schedule create --cron 'M H * * *' --tz ZONE --name NAME --gpu GPU --max-runtime 2h [--volume ID] -- COMMAND\nruncrate schedule list | runcrate schedule delete ID\nruncrate volumes create --name NAME --size GB --region REGION | runcrate volumes list | runcrate volumes delete ID\nruncrate billing balance | runcrate billing usage | runcrate secrets list | runcrate mcp install --client cursor\nGPUs: H100, H200, B200, A100, A100_80G, L40S, L40, A6000, A4000, RTX4000Ada. Times like 90m, 2h, 1h30m.\"\"\"\n\n\n# \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 on the GPU: train, serve, predict \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\ndef job() -> None:\n import threading, urllib.request\n import torch\n from peft import LoraConfig, get_peft_model\n from transformers import AutoModelForCausalLM, AutoTokenizer\n\n out = Path(os.environ.get(\"RUNCRATE_OUTPUT_DIR\", \"out\")); out.mkdir(parents=True, exist_ok=True)\n train = [json.loads(l) for l in open(\"train.jsonl\")]\n evals = [json.loads(l) for l in open(\"eval.jsonl\")]\n marks = {\"gpu\": torch.cuda.get_device_name(0)}\n print(f\"{marks['gpu']}: {len(train)} train, {len(evals)} eval\", flush=True)\n\n t0 = time.time()\n tok = AutoTokenizer.from_pretrained(BASE)\n model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16).cuda()\n model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type=\"CAUSAL_LM\",\n target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\", \"gate_proj\", \"up_proj\", \"down_proj\"]))\n rows = []\n for ex in train: # loss on the answer only\n prompt = tok.apply_chat_template([{\"role\": \"system\", \"content\": SYSTEM}, {\"role\": \"user\", \"content\": ex[\"request\"]}], tokenize=False, add_generation_prompt=True)\n p_ids = tok(prompt, add_special_tokens=False).input_ids\n a_ids = tok(ex[\"command\"] + tok.eos_token, add_special_tokens=False).input_ids\n rows.append((p_ids + a_ids, [-100] * len(p_ids) + a_ids))\n opt = torch.optim.AdamW(model.parameters(), lr=2e-4)\n model.train()\n for epoch in range(3):\n random.Random(epoch).shuffle(rows)\n for i in range(0, len(rows), 8):\n batch = rows[i:i + 8]\n n = max(len(r[0]) for r in batch)\n ids = torch.tensor([r[0] + [tok.pad_token_id] * (n - len(r[0])) for r in batch]).cuda()\n labels = torch.tensor([r[1] + [-100] * (n - len(r[1])) for r in batch]).cuda()\n loss = model(input_ids=ids, attention_mask=(ids != tok.pad_token_id).long(), labels=labels).loss\n loss.backward(); opt.step(); opt.zero_grad()\n print(f\"epoch {epoch + 1} loss {loss.item():.4f}\", flush=True)\n model.save_pretrained(out / \"adapter\")\n marks[\"train_seconds\"] = round(time.time() - t0, 1)\n del model, opt; torch.cuda.empty_cache()\n\n # Serve the base and the adapter from one vLLM server, as you would on an instance.\n t0 = time.time()\n server = subprocess.Popen([sys.executable, \"-m\", \"vllm.entrypoints.openai.api_server\", \"--model\", BASE, \"--port\", \"8000\",\n \"--enable-lora\", \"--lora-modules\", f\"cli={out / 'adapter'}\", \"--max-lora-rank\", \"16\",\n \"--max-model-len\", \"1024\", \"--gpu-memory-utilization\", \"0.5\", \"--enforce-eager\"],\n stdout=open(out / \"vllm.log\", \"w\"), stderr=subprocess.STDOUT)\n for _ in range(300):\n try:\n urllib.request.urlopen(\"http://127.0.0.1:8000/health\", timeout=2); break\n except Exception:\n if server.poll() is not None:\n raise RuntimeError(\"vLLM exited: see vllm.log\")\n time.sleep(1)\n marks[\"vllm_ready_seconds\"] = round(time.time() - t0, 1)\n\n def ask(model_name: str, request: str) -> str:\n body = json.dumps({\"model\": model_name, \"temperature\": 0, \"max_tokens\": 96,\n \"messages\": [{\"role\": \"system\", \"content\": SYSTEM}, {\"role\": \"user\", \"content\": request}]}).encode()\n req = urllib.request.Request(\"http://127.0.0.1:8000/v1/chat/completions\", data=body, headers={\"Content-Type\": \"application/json\"})\n return json.loads(urllib.request.urlopen(req, timeout=60).read())[\"choices\"][0][\"message\"][\"content\"].strip()\n\n t0 = time.time()\n preds = []\n with ThreadPoolExecutor(max_workers=16) as pool:\n for name in (BASE, \"cli\"):\n for ex, got in zip(evals, pool.map(lambda e: ask(name, e[\"request\"]), evals)):\n preds.append({\"model\": \"base\" if name == BASE else \"lora\", **ex, \"prediction\": got})\n marks[\"predict_seconds\"] = round(time.time() - t0, 1)\n server.terminate()\n (out / \"predictions.jsonl\").write_text(\"\".join(json.dumps(p) + \"\\n\" for p in preds))\n (out / \"marks.json\").write_text(json.dumps(marks))\n print(json.dumps(marks), flush=True)\n\n\nif __name__ == \"__main__\":\n job()\n";
const api = new OpenAI({ baseURL: "https://api.runcrate.ai/v1", apiKey: process.env.RUNCRATE_API_KEY, maxRetries: 4 });
const rc = new Runcrate({ apiKey: process.env.RUNCRATE_API_KEY });
type Pair = { request: string; command: string };
type Usage = { cost?: number };
const ledger: Record = {};
// 1. Synthetic data: varied requests, one correct command each.
let t0 = Date.now();
const schema = {
type: "object", additionalProperties: false, required: ["pairs"],
properties: { pairs: { type: "array", items: { type: "object", additionalProperties: false, required: ["request", "command"], properties: { request: { type: "string" }, command: { type: "string" } } } } },
};
const batches = await Promise.all(
Array.from({ length: 12 }, async (_, seed) => {
const r = await api.chat.completions.create({
model: TEACHER, temperature: 1, seed,
response_format: { type: "json_schema", json_schema: { name: "pairs", strict: true, schema } },
messages: [{ role: "user", content: `CLI reference:\n${CHEATSHEET}\n\nWrite 20 different requests a developer might type, casual or terse, each with the single correct command. Vary GPUs, names, times, flags. Batch ${seed}.` }],
});
return { pairs: JSON.parse(r.choices[0].message.content ?? "{}").pairs as Pair[], cost: (r.usage as Usage | undefined)?.cost ?? 0 };
}),
);
const seen = new Set();
const pairs: Pair[] = [];
let dataCost = 0;
for (const b of batches) {
dataCost += b.cost;
for (const p of b.pairs) if (p.command.startsWith("runcrate ") && !seen.has(p.request.toLowerCase())) (seen.add(p.request.toLowerCase()), pairs.push(p));
}
pairs.sort(() => Math.random() - 0.5);
const evals = pairs.slice(0, 40);
const train = pairs.slice(40);
ledger.data = { seconds: (Date.now() - t0) / 1000, cost_usd: dataCost, train: train.length, eval: evals.length };
console.log(`data: ${train.length} train, ${evals.length} eval, $${dataCost.toFixed(6)}`);
// 2. The job: job.py and the data, packed as a tar.gz, on the first GPU that is free.
const code = mkdtempSync(join(tmpdir(), "lora-"));
writeFileSync(join(code, "recipe.py"), JOB_PY);
writeFileSync(join(code, "train.jsonl"), train.map((p) => `${JSON.stringify(p)}\n`).join(""));
writeFileSync(join(code, "eval.jsonl"), evals.map((p) => `${JSON.stringify(p)}\n`).join(""));
const archive = join(tmpdir(), `lora-${Date.now()}.tar.gz`);
execFileSync("tar", ["-czf", archive, "-C", code, "."]);
const spec = {
command: "pip install -q peft && python3 recipe.py --job", gpuType: GPUS, template: "ubuntu-inference",
maxRuntimeMinutes: 10, maxRetries: 1, name: "lora-cli-recipe", tags: { recipe: "lora-finetune-serve-eval" },
};
if (process.argv.includes("--dry-run")) {
console.log(JSON.stringify({ jobs_run: spec, upload: { archive, bytes: readFileSync(archive).length } }, null, 2));
process.exit(0);
}
const upload = await rc.jobs.uploads.upload(archive);
t0 = Date.now();
const submitted = await rc.jobs.run({ ...spec, code: { uploadId: upload.uploadId } });
console.log(`job ${submitted.id} submitted`);
for await (const chunk of rc.jobs.followLogs(submitted.id)) process.stdout.write(chunk.text);
const done = await rc.jobs.wait(submitted.id, { timeout: 1800 });
const jobCost = done.attempts.reduce((s, a) => s + (a.costUsd ?? 0), 0);
console.log(`\njob ${done.status} on ${done.gpuType}, ${((Date.now() - t0) / 1000).toFixed(1)}s from submit, $${jobCost.toFixed(4)}`);
if (done.status !== "succeeded") throw new Error(`job ended ${done.status}: ${done.error}`);
mkdirSync("out", { recursive: true });
for (const a of await rc.jobs.artifacts.list(done.id)) {
mkdirSync(dirname(join("out", a.path)), { recursive: true });
await rc.jobs.artifacts.downloadTo(done.id, a.path, join("out", a.path));
}
const preds: (Pair & { model: string; prediction: string; correct?: boolean })[] = readFileSync("out/predictions.jsonl", "utf8").trim().split("\n").map((l) => JSON.parse(l));
// 3. Score: exact match, then a judge for commands that differ only in form.
const norm = (s: string) => s.trim().replace(/^`+|`+$/g, "").replace(/"/g, "'").replace(/\s+/g, " ");
let scoreCost = 0;
await Promise.all(
preds.map(async (p) => {
if (norm(p.prediction) === norm(p.command)) return void (p.correct = true);
const r = await api.chat.completions.create({
model: JUDGE, temperature: 0,
response_format: { type: "json_schema", json_schema: { name: "v", strict: true, schema: { type: "object", additionalProperties: false, required: ["same"], properties: { same: { type: "boolean" } } } } },
messages: [{ role: "user", content: `CLI reference:\n${CHEATSHEET}\n\nRequest: ${p.request}\nReference: ${p.command}\nCandidate: ${p.prediction}\nDoes the candidate do exactly what the request asks, as the reference does? Flag order does not matter.` }],
});
scoreCost += (r.usage as Usage | undefined)?.cost ?? 0;
p.correct = JSON.parse(r.choices[0].message.content ?? "{}").same;
}),
);
for (const m of ["base", "lora"]) {
const mine = preds.filter((p) => p.model === m);
console.log(`${m.padEnd(5)} ${mine.filter((p) => p.correct).length}/${mine.length} correct (${mine.filter((p) => norm(p.prediction) === norm(p.command)).length} exact)`);
}
writeFileSync("run.json", JSON.stringify({ ...ledger, job: { id: done.id, status: done.status, cost_usd: jobCost }, score_usd: scoreCost, total_usd: dataCost + jobCost + scoreCost }, null, 2));
```
## What the proof run cost
The Python file end to end: 188 teacher-written pairs (148 train, 40 held out), one GPU job that trained, served and predicted, then the judge. Measured on 2026-10-23 against `api.runcrate.ai`; costs are what the account was billed, read from each response's `X-LLM-Cost` header or `usage.cost`.
| Step | Model or tool | Time | Cost |
| - | - | - | - |
| Write 12 batches of pairs | openai/gpt-oss-120b (free tier) | 13.3 s | \$0.0000 |
| Job: boot, pip install peft, model download | A6000 job (ubuntu-inference) | about 6 min | in the job cost |
| Train LoRA r=16, 3 epochs | Qwen2.5-0.5B-Instruct on the A6000 | 27.1 s | in the job cost |
| vLLM up with base and adapter | vLLM on the same GPU | 53.1 s | in the job cost |
| 80 predictions (40 base, 40 LoRA) | vLLM | 25.3 s | in the job cost |
| Whole job, submit to finish | RTX A6000, \$0.74/hr listed | 472.5 s | \$0.0500 |
| Judge the 47 answers that were not exact | openai/gpt-oss-120b | 9.6 s | \$0.0006 |
| Total | | 510 s | \$0.0506 |
Result on the 40 held-out requests: the base model got 0 right (it answered in prose and reached for other tools' commands), the LoRA got 33 right, all 33 exact. The adapter is 34 MB; a job's outputs are kept 30 days (this proof's job was deleted afterwards with its outputs). The TypeScript file was run with `--dry-run` (data written, archive packed, the job spec printed, nothing submitted) to stay within the proof's GPU budget; it writes the same `recipe.py` the Python run used.
## Scale it
* Serve the adapter: start an [instance](https://runcrate.ai/docs/instances/overview.md) on the Inference template with `vllm serve Qwen/Qwen2.5-0.5B-Instruct --enable-lora --lora-modules cli=/workspace/adapter`, the adapter on a [volume](https://runcrate.ai/docs/storage/overview.md), and an [HTTPS link](https://runcrate.ai/docs/instances/https-links.md) to port 8000.
* More data: the teacher batches run in parallel; thousands of pairs take a minute.
* A sweep: `params` on the job runs one training per learning rate or rank as a group, with `max_parallel`.
* Safe retries: `max_retries: 1` restarts on a new machine if the first one fails to start.
## Gotchas found in the proof run
* The job spent about 6 of its 8 minutes before the command started: machine boot, the driver, and the template's first start. Budget a few minutes of start-up into `max_runtime_minutes`; it counts from the command, not the boot.
* vLLM 0.11 with `--enforce-eager` was up in 53 s for a 0.5B model; without it, CUDA graph capture adds time on every start.
* Free the training model before starting vLLM in the same process tree (`del model; torch.cuda.empty_cache()`) and give vLLM half the memory (`--gpu-memory-utilization 0.5`).
* Several teacher pairs were near-duplicates; deduplicate on the request text before splitting, or the held-out set leaks into training.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.