> ## Documentation Index
> Fetch the complete documentation index at: https://runcrate.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Jobs

> Run one command on its own GPU machine, follow its logs, and stop paying the moment it exits.

export const RuncrateStyles = () => {
  if (typeof document !== 'undefined' && !document.getElementById('runcrate-overrides')) {
    const s = document.createElement('style');
    s.id = 'runcrate-overrides';
    s.textContent = `
      /* Match Runcrate's rounding scale (--radius: 0.75rem) */
      .rounded-sm { border-radius: 0.5rem !important; }   /* 8px */
      .rounded-md { border-radius: 0.625rem !important; } /* 10px */
      .rounded-lg { border-radius: 0.75rem !important; }  /* 12px */
      .rounded-l-sm { border-top-left-radius: 0.5rem !important; border-bottom-left-radius: 0.5rem !important; }
      .rounded-r-sm { border-top-right-radius: 0.5rem !important; border-bottom-right-radius: 0.5rem !important; }
      .rounded-l-md { border-top-left-radius: 0.625rem !important; border-bottom-left-radius: 0.625rem !important; }
      .rounded-r-md { border-top-right-radius: 0.625rem !important; border-bottom-right-radius: 0.625rem !important; }
      .rounded-l-lg { border-top-left-radius: 0.75rem !important; border-bottom-left-radius: 0.75rem !important; }
      .rounded-r-lg { border-top-right-radius: 0.75rem !important; border-bottom-right-radius: 0.75rem !important; }

      /* Cards: never pure white in light mode */
      .card { background-color: #fcfcfc !important; border-radius: 0.75rem !important; }
      html.dark .card { background-color: #141414 !important; }

      /* Docs hero box */
      .rc-hero { background-color: #fcfcfc; border: 1px solid #e0e0e0; }
      html.dark .rc-hero { background-color: #141414; border-color: #242424; }
      html.dark .rc-hero h1 { color: #f5f5f5; }

      /* Runcrate scrollbar — thin, transparent track, hide-until-hover thumb */
      ::-webkit-scrollbar { width: 6px; height: 6px; background-color: transparent; }
      ::-webkit-scrollbar-track { background-color: transparent; }
      ::-webkit-scrollbar-thumb { background-color: rgba(155, 155, 155, 0.5); border-radius: 10px; transition: opacity 0.3s ease; opacity: 0; }
      ::-webkit-scrollbar-thumb:hover { background-color: rgba(155, 155, 155, 0.7); }
      *:hover::-webkit-scrollbar-thumb,
      *:focus::-webkit-scrollbar-thumb,
      *:active::-webkit-scrollbar-thumb { opacity: 1; }
      * { scrollbar-width: thin; scrollbar-color: rgba(155, 155, 155, 0.5) transparent; }
    `;
    document.head.appendChild(s);
  }
  return null;
};

<RuncrateStyles />

A job is one command on its own fresh GPU machine. Runcrate starts the machine, runs your command, streams its output, and stops the machine as soon as the command exits. Billing ends with it.

Every job has a maximum runtime, so a job never runs unbounded. You do not have to remember to shut anything down.

## How a job runs

<Steps>
  <Step title="Submit">
    You send a command, a GPU type and a maximum runtime. The job is `queued` while a machine is requested.
  </Step>

  <Step title="Start">
    A fresh machine with that GPU launches and runs its setup (`starting`). If you attached a volume, its files are in `/workspace`. Some machines install the GPU driver after they boot: the job waits for it, so your command starts with the GPU ready.
  </Step>

  <Step title="Run">
    Your command runs under bash in `/workspace` (`running`). Its stdout and stderr are streamed as logs you can follow live.
  </Step>

  <Step title="Stop">
    When the command exits, hits its maximum runtime or is cancelled, the machine stops and billing ends. Files under `/workspace` are saved back to the volume.
  </Step>
</Steps>

While the job runs, its machine also shows in **Instances**. Stopping it there cancels the job.

## Run a job

<Tabs>
  <Tab title="Ask your agent">
    Connect the [Runcrate MCP server](/docs/mcp/setup) to Claude Code, Codex, Cursor or any MCP client, then ask in plain words:

    ```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
    Run python train.py on an A4000 for at most 2 hours with my volume training-data, then tell me the final loss.
    ```

    ```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
    Fine-tune with ./scripts/finetune.sh on 2 H100s, cap it at 6 hours, pass my HF_TOKEN, and tell me if it fails.
    ```

    ```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
    What did my last job print before it stopped? Cancel it if it is still running.
    ```

    What the agent does with the first prompt:

    1. `list_volumes` to find the id of `training-data`.
    2. `run_job` with `gpu: "A4000"`, `command: "python train.py"`, `max_runtime_minutes: 120` and the volume's `storage_id`. It gets the job id back.
    3. `get_job_logs` with `follow: true`, passing the returned `cursor` back as `after` each time, until `done` is true. It reads the loss from the output.
    4. `get_job` for the final status, exit code and cost, then answers you.

    For the second prompt the agent passes the token in `run_job`'s `env` object (`{ "HF_TOKEN": "..." }`). For the third it finds the job with `list_jobs`, reads it with `get_job_logs`, and stops it with `cancel_job`. `run_job` also takes `region`, and `workspace` and `environment` to choose where the job runs. See [MCP tools](/docs/mcp/tools#jobs) for every parameter.
  </Tab>

  <Tab title="See the code">
    The CLI submits the job and follows its logs until it ends:

    ```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
    runcrate run --gpu A4000 --volume <volume-id> --max-runtime 2h -- python train.py
    ```

    The same job over the API:

    <CodeGroup>
      ```bash Submit theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
      curl -X POST https://www.runcrate.ai/api/v1/jobs \
        -H "Authorization: Bearer $RUNCRATE_API_KEY" \
        -H "Content-Type: application/json" \
        -d '{
          "command": "python train.py",
          "gpu_type": "A4000",
          "max_runtime_minutes": 120,
          "storage_id": "<volume-id>",
          "env": { "HF_TOKEN": "hf_..." }
        }'
      ```

      ```bash Follow logs theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
      cursor=0
      while :; do
        page=$(curl -s "https://www.runcrate.ai/api/v1/jobs/$JOB_ID/logs?after=$cursor&follow=true" \
          -H "Authorization: Bearer $RUNCRATE_API_KEY")
        echo "$page" | jq -j '.data.chunks[].text'
        cursor=$(echo "$page" | jq '.data.cursor')
        [ "$(echo "$page" | jq '.data.done')" = "true" ] && break
      done
      ```

      ```bash Status theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
      curl https://www.runcrate.ai/api/v1/jobs/$JOB_ID \
        -H "Authorization: Bearer $RUNCRATE_API_KEY"
      ```

      ```bash Cancel theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
      curl -X POST https://www.runcrate.ai/api/v1/jobs/$JOB_ID/cancel \
        -H "Authorization: Bearer $RUNCRATE_API_KEY"
      ```
    </CodeGroup>

    In Python, submit and print the output as it arrives:

    ```python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
    import os
    import requests

    API = "https://www.runcrate.ai/api/v1"
    headers = {"Authorization": f"Bearer {os.environ['RUNCRATE_API_KEY']}"}

    r = requests.post(f"{API}/jobs", headers=headers, json={
        "command": "python train.py",
        "gpu_type": "A4000",
        "max_runtime_minutes": 120,
        "storage_id": "<volume-id>",
    })
    r.raise_for_status()
    job_id = r.json()["data"]["id"]

    cursor = 0
    while True:
        # follow=true waits up to 25 s for new output, so no sleep is needed
        r = requests.get(f"{API}/jobs/{job_id}/logs", headers=headers,
                         params={"after": cursor, "follow": "true"}, timeout=60)
        r.raise_for_status()
        logs = r.json()["data"]
        for chunk in logs["chunks"]:
            print(chunk["text"], end="")
        cursor = logs["cursor"]
        if logs["done"]:
            break

    job = requests.get(f"{API}/jobs/{job_id}", headers=headers).json()["data"]
    print(job["status"], job["exit_code"], job["cost_usd"])
    ```

    The full request and response shapes are in the [API reference](/docs/api-reference).
  </Tab>
</Tabs>

## Statuses

| Status | Meaning |
| - | - |
| `queued` | Accepted. A machine is being requested. |
| `starting` | The machine is launching and running its setup, waiting for the GPU driver, and pulling the `image` if you set one. None of this counts against the maximum runtime. |
| `running` | Your command has started. `started_at` is set. |
| `succeeded` | The command exited with code 0. |
| `failed` | The command exited non-zero (`exit_code` is set), or the machine failed or was stopped (`exit_code` is null and `error` says why, for example the budget was reached or the balance ran out). |
| `cancelled` | Cancelled through the API, CLI or MCP, or the machine was stopped from Instances. |
| `timed_out` | The command hit its maximum runtime. |

`succeeded`, `failed`, `cancelled` and `timed_out` are final: they never change, and `ended_at` is set on all of them.

## Logs

Output is stdout and stderr combined, in order, delivered as chunks. A chunk is a raw fragment of output, not a line: print its `text` as is.

* **Follow with a cursor.** Each response has a `cursor`. Pass it back as `after` to get only what is new. With `follow=true`, a request waits up to 25 seconds for new output or for the job to end.
* **Stop on `done`.** `done: true` means the job has ended and you have read every chunk.
* **Capped at 10 MB per job.** The first 5 MB and the newest output are kept. When the middle is dropped, `truncated` is true and a marker chunk with `seq: null` sits where the gap is.
* **Very fast output.** When a command writes faster than its output can be sent, the newest output is sent first and the part in between is dropped the same way, so a job's final lines are never held back.
* **Kept 30 days** after the job ends.

## Results and volumes

Attach a volume with `storage_id` (CLI: `--volume`) and its files are in `/workspace` when your command starts. Write checkpoints, metrics and outputs under `/workspace`: they are saved back to the volume when the machine stops, whatever the final status. Without a volume, files on the machine are gone when it stops, so print what you need to the logs.

The volume must be in the same workspace and environment as the job, and not attached to another machine. See [Storage](/docs/storage/overview).

## Environment variables

Every job gets its environment's secrets, set in the dashboard, as environment variables, along with `RUNCRATE_ENVIRONMENT`, `RUNCRATE_INSTANCE_ID` and `RUNCRATE_JOB_ID`. Pass anything else, such as a one-off `HF_TOKEN`, in `env`; a name set in both takes the value from `env`. The values reach your command, are stored only encrypted and are never shown again: the job only reports their names in `env_keys`. If the machine cannot get them when it starts, the command does not run and the job fails. Sensitive hides a secret's value in Runcrate. Every machine launched into its environments receives it, so anyone who can launch there can read it on the machine.

## Containers

Set `image` to run your command inside a container image instead of on the template directly. `/workspace` is mounted, and the image is pulled while the job is `starting`, so the pull does not use up the maximum runtime. The image needs `bash` or `sh`.

The GPUs are passed into the container only on machines that have the NVIDIA container runtime. On other machines the job log says so and the command runs in the container without GPUs. If your command needs the GPU, leave `image` out: on the machine itself the GPUs are always available.

## Workspaces and environments

Jobs belong to a workspace and an environment, like instances. An API key or agent session sees only the jobs in its own environment. Otherwise, pick one with the `X-Environment` header or the `environment` query parameter (MCP: the `environment` parameter).

## Billing

A job is billed at its machine's hourly rate (`cost_per_hour`, in dollars) from when the machine is running until it stops. Setup time is included. `cost_usd` on the job is the cost so far and is final once the job ends. See [pricing](/docs/billing/pricing) for current rates per GPU-hour.

The maximum runtime is counted from when your command starts, so setup does not use it up.

## GPU readiness

Your command starts once the GPU driver answers. On machines that install the driver after they boot, the job stays `starting` while it waits, for up to 20 minutes. If the driver is still not ready after that, the command runs anyway, the job log says so, and GPU calls in it will usually fail, so the job usually ends `failed`. Cancel and resubmit if that happens; a new job gets a fresh machine.

## Limits

| Limit | Value |
| - | - |
| Maximum runtime | 1 minute to 24 hours, required on every job |
| GPUs per job | 1 to 8 |
| Command | 64 KB |
| Environment variables | 200 keys, 32 KB per value, 64 KB in total with the environment's secrets |
| Log size | 10 MB per job (first 5 MB and newest output kept) |
| Log retention | 30 days after the job ends |
| Log long-poll | Up to 25 seconds per request |

## Errors

| Code | Status | When |
| - | - | - |
| `validation_error` | 422 | A field is missing or out of range. |
| `BUDGET_REACHED` | 402 | The environment's monthly budget is spent. Checked before any machine is requested. |
| `insufficient_credits` | 402 | Your balance cannot cover the job. |
| `bad_request` | 400 | The volume cannot be used, for example it is attached to another machine. |
| `not_found` | 404 | The job or volume does not exist in this workspace and environment. |
| `conflict` | 409 | You cancelled a job that already ended, or another launch claimed the volume first. |
| `service_unavailable` | 503 | Jobs are briefly unavailable. Retry shortly. |

If the machine cannot be launched after the job was created, the error carries `details.job_id` and that job is `failed`.

## When to use an instance instead

A job fits work that starts, runs and ends on its own: training runs, evaluations, batch inference, data processing. Use an [instance](/docs/instances/overview) when you need to work on the machine yourself (SSH, Jupyter, VS Code), keep a server up for others to call, or run longer than 24 hours.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.