Skip to main content
A popular GPU is sometimes sold out for a few minutes. A job does not have to fail when that happens: give it more than one GPU it can run on, and let it wait.

​
A list of GPUs

gpu_type takes one GPU or a list. Runcrate tries them in the order you give, and runs the job on the first one with capacity. An entry can carry its own GPU count after a colon.
  • "A100-80GB:2" means two A100 80GB GPUs. Entries without a count use gpu_count (default 1).
  • Each GPU is tried on several machine options before the next one, cheapest first.
  • gpu_candidates on the job is the list you asked for. gpu_type and gpu_count become what it actually ran on, so the cost and the logs match the machine.

​
Waiting for capacity

When none of the GPUs has capacity, the job stays queued with queue_reason: "no_capacity" and is tried again after 1, 2 and 4 minutes, then every 5 minutes. next_attempt_at says when. It waits up to capacity_wait_minutes (0 to 1440, default 30), then fails with an error that says what it tried:
Set capacity_wait_minutes: 0 to fail at once instead, as a job did before waiting existed. A machine option that just failed to launch rests for 10 minutes, so a wait never hammers one that is out of stock. A waiting job costs nothing. cancel_job (or POST /jobs/{id}/cancel) stops waiting at once. The workspace gets a job.queued webhook event when a job starts waiting.

​
Ask your agent

The agent calls run_job with gpu: ["H100", "A100-80GB:2"] and capacity_wait_minutes: 60, then follows the logs once the job is running. get_job shows queue_reason and next_attempt_at while it waits.