> ## Documentation Index
> Fetch the complete documentation index at: https://runcrate.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Multilingual video dubbing

> Dub a video into other languages: transcribe, translate to fit the timing, voice each line, check the dub by ear, and mux.

A product video in one language reaches a fraction of its audience. Dubbing is a timing problem more than a translation problem: each line has to fit the seconds the original took. This pipeline translates with the seconds in the prompt, voices each line, speeds a long line up by at most 35%, and then listens to its own work: it transcribes every dubbed line and flags the ones that do not say what they should. Two languages for a 7-second ad cost 2 cents.

```mermaid theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
flowchart LR
  A[video.mp4] --> B[Segments<br/>whisper-large-v3-turbo]
  B --> C[Translate to fit<br/>gpt-oss-120b]
  C --> D[Voice each line<br/>Qwen3-TTS]
  D --> E[Fit the slot<br/>atempo up to 1.35x]
  E --> F[Check by ear<br/>whisper, word match]
  E --> G[ffmpeg: video.es.mp4 + .srt]
```

## Build it in your project

Paste this line into your agent (Claude Code, Codex, Cursor, OpenCode, Cline). It sets up your key, then builds the pipeline in your project:

```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
Add Runcrate's Qwen3 TTS to this project: run curl -fsSL "https://runcrate.ai/agents.md?model=Qwen%2FQwen3-TTS&intent=build" and follow it. Then build: a script that dubs a video into the languages given (es,de...): whisper-large-v3-turbo segments, gpt-oss-120b translation that fits each segment's seconds, Qwen3-TTS per line fitted with atempo, a back-transcription check that flags lines under 85% word match, subtitles and an ffmpeg mux.
```

## The code

One file per language. Both read `RUNCRATE_API_KEY` from the environment; nothing secret is in the code.

```python video_dubbing.py theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
"""Dub a video into other languages: transcribe, translate to fit the timing, voice each line, check the dub by ear, mux.

pip install openai      (and ffmpeg on PATH)
RUNCRATE_API_KEY=... python video_dubbing.py video.mp4 es,de
"""
import difflib, json, os, re, subprocess, sys, time
from pathlib import Path

from openai import OpenAI

client = OpenAI(base_url="https://api.runcrate.ai/v1", api_key=os.environ["RUNCRATE_API_KEY"], max_retries=4, timeout=180)
ASR = "openai/whisper-large-v3-turbo"
TRANSLATOR = "openai/gpt-oss-120b"          # on the free tier
TTS, VOICE = "Qwen/Qwen3-TTS", "Serena"      # speaks the language of the text it is given
LANGS = {"es": "Spanish", "de": "German", "fr": "French", "it": "Italian", "pt": "Portuguese", "ja": "Japanese", "ko": "Korean", "zh": "Chinese"}
OUT = Path("out"); OUT.mkdir(exist_ok=True)
ledger = []

def log(step, model, t0, cost):
    ledger.append({"step": step, "model": model, "seconds": round(time.time() - t0, 2), "cost_usd": cost})
    print(f"{step:<14} {model:<34} {time.time() - t0:7.2f}s  ${cost:.6f}")

def ffmpeg(*args):
    subprocess.run(["ffmpeg", "-y", "-loglevel", "error", *args], check=True)

def duration(path: Path) -> float:
    return float(subprocess.run(["ffprobe", "-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", str(path)], capture_output=True, text=True).stdout)

def srt_time(t: float) -> str:
    return f"{int(t // 3600):02d}:{int(t % 3600 // 60):02d}:{int(t % 60):02d},{int(t * 1000 % 1000):03d}"

def words(s: str) -> list[str]:
    return re.findall(r"\w+", s.lower())

def main(src: str, langs: list[str]) -> None:
    src = Path(src)
    ffmpeg("-i", str(src), "-vn", "-ac", "1", "-ar", "16000", str(OUT / "source.wav"))
    t0 = time.time()
    with open(OUT / "source.wav", "rb") as f:
        raw = client.audio.transcriptions.with_raw_response.create(model=ASR, file=f, response_format="verbose_json")
    segs = [{"start": s.start, "end": s.end, "text": s.text.strip()} for s in raw.parse().segments]
    log("transcribe", ASR, t0, float(raw.headers.get("x-llm-cost", 0)))
    report = {}

    for lang in langs:
        name = LANGS[lang]
        t0 = time.time()
        raw = client.chat.completions.with_raw_response.create(
            model=TRANSLATOR, temperature=0,
            messages=[{"role": "user", "content": f"Translate each line into natural spoken {name} for a voice-over. Each line must be speakable "
                       f"in the seconds given, so shorten rather than overrun. Keep product names.\n\n"
                       + "\n".join(f"{i}. ({s['end'] - s['start']:.1f}s) {s['text']}" for i, s in enumerate(segs))}],
            response_format={"type": "json_schema", "json_schema": {"name": "lines", "strict": True, "schema": {
                "type": "object", "additionalProperties": False, "required": ["lines"],
                "properties": {"lines": {"type": "array", "items": {"type": "string"}}}}}},
        )
        lines = json.loads(raw.parse().choices[0].message.content)["lines"][:len(segs)]
        log(f"translate {lang}", TRANSLATOR, t0, float(raw.headers.get("x-llm-cost", 0)))

        parts, check = [], []
        for i, (seg, line) in enumerate(zip(segs, lines)):
            t0 = time.time()
            raw = client.audio.speech.with_raw_response.create(model=TTS, voice=VOICE, input=line, response_format="wav")
            clip = OUT / f"{lang}-{i:03d}.wav"; clip.write_bytes(raw.content)
            cost = float(raw.headers.get("x-llm-cost", 0))
            # Fit the slot: speed up to 1.35x if the line runs long (atempo keeps the pitch).
            slot = max(0.5, (segs[i + 1]["start"] if i + 1 < len(segs) else seg["end"] + 1.0) - seg["start"])
            speed = min(1.35, max(1.0, duration(clip) / slot))
            fitted = OUT / f"{lang}-{i:03d}-fit.wav"
            ffmpeg("-i", str(clip), "-filter:a", f"atempo={speed:.3f}", str(fitted))
            parts.append((seg["start"], fitted))
            # Check by ear: transcribe the dub and compare it with the line it should say.
            with open(fitted, "rb") as f:
                raw2 = client.audio.transcriptions.with_raw_response.create(model=ASR, file=f, language=lang)
            cost += float(raw2.headers.get("x-llm-cost", 0))
            heard = raw2.parse().text
            match = difflib.SequenceMatcher(None, words(line), words(heard)).ratio()
            check.append({"line": line, "heard": heard.strip(), "match": round(match, 2), "speed": round(speed, 2)})
            log(f"voice {lang} {i}", TTS, t0, cost)

        t0 = time.time()
        inputs, mix = [], []
        for k, (start, path) in enumerate(parts):
            inputs += ["-i", str(path)]
            mix.append(f"[{k + 1}:a]adelay={int(start * 1000)}:all=1[d{k}]")
        track = OUT / f"dub-{lang}.wav"
        ffmpeg("-f", "lavfi", "-t", f"{duration(src)}", "-i", "anullsrc=r=24000:cl=mono", *inputs, "-filter_complex",
               ";".join(mix) + ";[0:a]" + "".join(f"[d{k}]" for k in range(len(parts))) + f"amix=inputs={len(parts) + 1}:normalize=0[a]",
               "-map", "[a]", str(track))
        srt = "".join(f"{i + 1}\n{srt_time(s['start'])} --> {srt_time(s['end'])}\n{l}\n\n" for i, (s, l) in enumerate(zip(segs, lines)))
        (OUT / f"{src.stem}.{lang}.srt").write_text(srt)
        ffmpeg("-i", str(src), "-i", str(track), "-map", "0:v", "-map", "1:a", "-c:v", "copy", "-c:a", "aac", "-shortest", str(OUT / f"{src.stem}.{lang}.mp4"))
        log(f"mux {lang}", "ffmpeg", t0, 0.0)

        report[lang] = {"lines": check, "min_match": min(c["match"] for c in check)}
        flag = [c for c in check if c["match"] < 0.85]
        print(f"  {name}: {len(check)} lines, lowest match {report[lang]['min_match']:.2f}" + (f", {len(flag)} to review" if flag else ""))

    total = sum(s["cost_usd"] for s in ledger)
    Path("run.json").write_text(json.dumps({"segments": segs, "check": report, "steps": ledger, "total_usd": total}, indent=2, ensure_ascii=False))
    print(f"\ntotal ${total:.4f}")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2].split(","))
```

```typescript video-dubbing.ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
// Dub a video into other languages: transcribe, translate to fit the timing, voice each line, check the dub by ear, mux.
//
// npm install openai      (and ffmpeg on PATH)
// RUNCRATE_API_KEY=... npx tsx video-dubbing.ts video.mp4 es,de
import { execFileSync } from "node:child_process";
import { createReadStream, mkdirSync, writeFileSync } from "node:fs";
import { parse } from "node:path";
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://api.runcrate.ai/v1", apiKey: process.env.RUNCRATE_API_KEY, maxRetries: 4, timeout: 180_000 });
const ASR = "openai/whisper-large-v3-turbo";
const TRANSLATOR = "openai/gpt-oss-120b"; // on the free tier
const TTS = "Qwen/Qwen3-TTS"; // speaks the language of the text it is given
const VOICE = "Serena";
const LANGS: Record<string, string> = { es: "Spanish", de: "German", fr: "French", it: "Italian", pt: "Portuguese", ja: "Japanese", ko: "Korean", zh: "Chinese" };
const ledger: { step: string; model: string; seconds: number; cost_usd: number }[] = [];
const ffmpeg = (...args: string[]) => execFileSync("ffmpeg", ["-y", "-loglevel", "error", ...args]);
const duration = (path: string) => Number(execFileSync("ffprobe", ["-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", path]).toString());
const words = (s: string) => s.toLowerCase().match(/[\p{L}\p{N}]+/gu) ?? [];
mkdirSync("out", { recursive: true });

function log(step: string, model: string, t0: number, cost: number) {
  const seconds = (Date.now() - t0) / 1000;
  ledger.push({ step, model, seconds, cost_usd: cost });
  console.log(`${step.padEnd(14)} ${model.padEnd(34)} ${seconds.toFixed(2).padStart(7)}s  $${cost.toFixed(6)}`);
}

function srtTime(t: number): string {
  const ms = Math.round(t * 1000);
  const p = (n: number, w = 2) => String(n).padStart(w, "0");
  return `${p(Math.floor(ms / 3_600_000))}:${p(Math.floor((ms % 3_600_000) / 60_000))}:${p(Math.floor((ms % 60_000) / 1000))},${p(ms % 1000, 3)}`;
}

/** How alike two word lists are, 0 to 1 (the share of words they have in common, in order). */
function similarity(a: string[], b: string[]): number {
  const dp = Array.from({ length: a.length + 1 }, () => new Array<number>(b.length + 1).fill(0));
  for (let i = 1; i <= a.length; i++) for (let j = 1; j <= b.length; j++) dp[i][j] = a[i - 1] === b[j - 1] ? dp[i - 1][j - 1] + 1 : Math.max(dp[i - 1][j], dp[i][j - 1]);
  return a.length + b.length ? (2 * dp[a.length][b.length]) / (a.length + b.length) : 1;
}

const [src, langArg] = process.argv.slice(2);
const stem = parse(src).name;
ffmpeg("-i", src, "-vn", "-ac", "1", "-ar", "16000", "out/source.wav");
let t0 = Date.now();
const tr = await client.audio.transcriptions.create({ model: ASR, file: createReadStream("out/source.wav"), response_format: "verbose_json" }).withResponse();
const segs = (tr.data.segments ?? []).map((s) => ({ start: s.start, end: s.end, text: s.text.trim() }));
log("transcribe", ASR, t0, Number(tr.response.headers.get("x-llm-cost") ?? 0));
const report: Record<string, { lines: { line: string; heard: string; match: number; speed: number }[]; min_match: number }> = {};

for (const lang of langArg.split(",")) {
  const name = LANGS[lang];
  t0 = Date.now();
  const res = await client.chat.completions
    .create({
      model: TRANSLATOR,
      temperature: 0,
      messages: [
        {
          role: "user",
          content:
            `Translate each line into natural spoken ${name} for a voice-over. Each line must be speakable ` +
            `in the seconds given, so shorten rather than overrun. Keep product names.\n\n` +
            segs.map((s, i) => `${i}. (${(s.end - s.start).toFixed(1)}s) ${s.text}`).join("\n"),
        },
      ],
      response_format: {
        type: "json_schema",
        json_schema: {
          name: "lines",
          strict: true,
          schema: { type: "object", additionalProperties: false, required: ["lines"], properties: { lines: { type: "array", items: { type: "string" } } } },
        },
      },
    })
    .withResponse();
  const lines: string[] = JSON.parse(res.data.choices[0].message.content ?? "{}").lines.slice(0, segs.length);
  log(`translate ${lang}`, TRANSLATOR, t0, Number(res.response.headers.get("x-llm-cost") ?? 0));

  const parts: [number, string][] = [];
  const check: { line: string; heard: string; match: number; speed: number }[] = [];
  for (const [i, line] of lines.entries()) {
    t0 = Date.now();
    const seg = segs[i];
    const speech = await client.audio.speech.create({ model: TTS, voice: VOICE, input: line, response_format: "wav" }).withResponse();
    const clip = `out/${lang}-${String(i).padStart(3, "0")}.wav`;
    writeFileSync(clip, Buffer.from(await speech.data.arrayBuffer()));
    let cost = Number(speech.response.headers.get("x-llm-cost") ?? 0);
    // Fit the slot: speed up to 1.35x if the line runs long (atempo keeps the pitch).
    const slot = Math.max(0.5, (i + 1 < segs.length ? segs[i + 1].start : seg.end + 1) - seg.start);
    const speed = Math.min(1.35, Math.max(1, duration(clip) / slot));
    const fitted = clip.replace(".wav", "-fit.wav");
    ffmpeg("-i", clip, "-filter:a", `atempo=${speed.toFixed(3)}`, fitted);
    parts.push([seg.start, fitted]);
    // Check by ear: transcribe the dub and compare it with the line it should say.
    const back = await client.audio.transcriptions.create({ model: ASR, file: createReadStream(fitted), language: lang }).withResponse();
    cost += Number(back.response.headers.get("x-llm-cost") ?? 0);
    check.push({ line, heard: back.data.text.trim(), match: Math.round(similarity(words(line), words(back.data.text)) * 100) / 100, speed: Math.round(speed * 100) / 100 });
    log(`voice ${lang} ${i}`, TTS, t0, cost);
  }

  t0 = Date.now();
  const track = `out/dub-${lang}.wav`;
  ffmpeg(
    "-f", "lavfi", "-t", `${duration(src)}`, "-i", "anullsrc=r=24000:cl=mono",
    ...parts.flatMap(([, p]) => ["-i", p]),
    "-filter_complex",
    parts.map(([start], k) => `[${k + 1}:a]adelay=${Math.trunc(start * 1000)}:all=1[d${k}]`).join(";") +
      `;[0:a]${parts.map((_, k) => `[d${k}]`).join("")}amix=inputs=${parts.length + 1}:normalize=0[a]`,
    "-map", "[a]", track,
  );
  writeFileSync(`out/${stem}.${lang}.srt`, segs.map((s, i) => `${i + 1}\n${srtTime(s.start)} --> ${srtTime(s.end)}\n${lines[i]}\n\n`).join(""));
  ffmpeg("-i", src, "-i", track, "-map", "0:v", "-map", "1:a", "-c:v", "copy", "-c:a", "aac", "-shortest", `out/${stem}.${lang}.mp4`);
  log(`mux ${lang}`, "ffmpeg", t0, 0);

  const minMatch = Math.min(...check.map((c) => c.match));
  report[lang] = { lines: check, min_match: minMatch };
  const flagged = check.filter((c) => c.match < 0.85).length;
  console.log(`  ${name}: ${check.length} lines, lowest match ${minMatch.toFixed(2)}${flagged ? `, ${flagged} to review` : ""}`);
}

const total = ledger.reduce((s, r) => s + r.cost_usd, 0);
writeFileSync("run.json", JSON.stringify({ segments: segs, check: report, steps: ledger, total_usd: total }, null, 2));
console.log(`\ntotal $${total.toFixed(4)}`);
```

## What the proof run cost

The Python file on the 7-second ad from the [video ad recipe](https://runcrate.ai/docs/examples/recipes/product-photo-to-video-ad.md), into Spanish and German. Measured on 2026-10-23 against `api.runcrate.ai`; costs are what the account was billed, read from each response's `X-LLM-Cost` header or `usage.cost`.

| Step | Model or tool | Time | Cost |
| - | - | - | - |
| Transcribe | openai/whisper-large-v3-turbo | 1.2 s | \$0.0025 |
| Translate, Spanish | openai/gpt-oss-120b (free tier) | 1.3 s | \$0.0000 |
| Voice and check two Spanish lines | Qwen/Qwen3-TTS + whisper | 6.5 s | \$0.0102 |
| Translate, German | openai/gpt-oss-120b (free tier) | 1.2 s | \$0.0000 |
| Voice and check two German lines | Qwen/Qwen3-TTS + whisper | 7.1 s | \$0.0079 |
| Two muxes | ffmpeg | 0.2 s | \$0 |
| Total | | 18 s | \$0.0205 |

The TypeScript file: \$0.0200 in 18 s. On that run the check flagged one line in each language: both were lines sped up to the 1.35x cap, heard back as "Montente" and "bleibe".

## Scale it

* Every language is independent: run them in parallel, or one [job](https://runcrate.ai/docs/jobs/overview.md) per language.
* Long videos: segments are short, so a 10-minute video is about 150 lines; the cost is the TTS characters plus two transcriptions of the audio.
* Review only what the check flags: the word-match score is in `run.json` for every line.

## Gotchas found in the proof run

* `ResembleAI/chatterbox-multilingual` mangled Spanish and German when given only the text ("sostenable Canoistra"); it takes no language parameter. Qwen3-TTS read the same lines cleanly, so the recipe uses it.
* A lip-synced talking head was tried and left out: `kuaishou/kling-avatar-v2` answered 502 three times, and Seedance 2.0 Fast refused a photographic portrait with reference audio as sensitive (E005). It goes in when it runs. When you add one: Kling Lip Sync and Kling Avatar v2 make a clip as long as the audio or video you send, so `duration` is ignored. Runcrate measures your input when you submit and needs that clip's cost free in your balance, with a second or so of margin; you're charged only for the clip's real length, and a balance too low is refused with `402` and `"code": "video_hold_unaffordable"`. See [video generation](https://runcrate.ai/docs/models/video-generation.md).
* Music under the voice is lost: this replaces the audio track. Separate the stems first (an open source-separation model on a GPU job) if the original has a score.
* The 1.35x cap is where speech starts to slur; the back-transcription check is how you find those lines.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.