> ## Documentation Index
> Fetch the complete documentation index at: https://runcrate.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Retries

> Run a job again on a new machine when its machine fails, or when the command fails with an exit code you choose.

Long jobs meet the occasional bad machine: the GPU driver never comes up, the machine goes silent, or it stops. With `max_retries`, Runcrate replaces the machine and runs the command again, so a multi-day training run is not lost to one failure.

## Turn it on

```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
  "command": "python train.py --resume-from /workspace/checkpoints",
  "gpu_type": "H100",
  "max_runtime_minutes": 2880,
  "storage_id": "<volume-id>",
  "max_retries": 3,
  "retry_exit_codes": [75]
}
```

| Field | Meaning |
| - | - |
| `max_retries` | 0 to 10 (default 0). How many more attempts after the first. |
| `retry_on` | `machine_failure` is always on once `max_retries` is above 0. Add `nonzero_exit` to also retry when the command fails. |
| `retry_exit_codes` | Retry only when the command exits with one of these codes. When set, it takes precedence over `nonzero_exit`: other codes end the job. |

What counts as a machine failure: the machine stopped or disappeared before the command finished, it never finished setting up, or Runcrate lost contact with the job for 10 minutes. A job that hit its maximum runtime, was cancelled, or was stopped by a budget or a low balance is never retried.

## What an attempt is

* Each attempt runs on a **new machine**, with the whole `max_runtime_minutes`: the maximum runtime counts per attempt.
* The volume is saved back when each attempt's machine stops, so the next attempt starts with the files the last one wrote. Write checkpoints under `/workspace` and resume from them.
* The command sees `RUNCRATE_JOB_ATTEMPT` (1, 2, ...).
* Between attempts the job is `queued` again, and `attempt` goes up by one. It waits for capacity like a new job, up to `capacity_wait_minutes`.
* The logs go on in one stream. A line marks where each new attempt starts:

```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
[runcrate] Attempt 2 of 4 on a new machine (previous: Lost contact with the job for 10 minutes)
```

Only the last attempt's outcome is the job's: `job.finished` goes out once, when the job really ends.

## Cost

`cost_usd` adds up every attempt's machine. `attempts` on the job lists each one with its machine, status, exit code, error, times, GPU, region and cost:

```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
"attempts": [
  { "attempt": 1, "deployment_id": "...", "status": "failed", "exit_code": null,
    "error": "The machine stopped before the command finished", "cost_usd": 12.4, "gpu_type": "H100" },
  { "attempt": 2, "deployment_id": "...", "status": "running", "cost_usd": 3.1, "gpu_type": "H100" }
]
```

## Ask your agent

```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
Fine-tune with ./finetune.sh on an H100 for up to 2 days, save checkpoints to my volume ckpts, and retry up to 3 times if the machine dies.
```

The agent calls `run_job` with `max_retries: 3` and the volume's `storage_id`, and tells your script to resume from the last checkpoint.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.