> ## Documentation Index > Fetch the complete documentation index at: https://runcrate.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Clusters > Several GPU machines on one private network, created, paid for and terminated as one, with Slurm, Kubernetes or Ray if you want them. A cluster is two or more GPU machines (its nodes) that you create, connect to, pay for and terminate as one thing. Every machine has the same GPUs, they share a private network and a `/shared` folder, and they reach each other over SSH by name (`node-0`, `node-1`, ...). Node 0 is the head. You pick the software that runs across the machines: | Software | What you get | | - | - | | **None** | The machines, the private network, a hostfile and ready-made `torchrun`, `mpirun` and DeepSpeed commands | | **Slurm** | A Slurm cluster: submit with `sbatch` and `srun`, see the queue with `squeue` | | **Kubernetes** | A Kubernetes cluster with GPU scheduling, and a kubeconfig you use from your own computer | | **Ray** | A Ray cluster with the head on node 0 and every node joined | See [Software](https://runcrate.ai/docs/clusters/software.md) for what each one installs and when it counts as ready. ## Bought by the machine A cluster is priced by the machine, never by the GPU. You pick a GPU and how many machines, and Runcrate shows the machines that can start now: each offer is a location, how the machines are joined, the kind of machine and the GPUs each one comes with (1, 2, 4 or 8), with the price per hour for the whole cluster and the price per machine under it. The whole cluster is the machine price times the number of machines. ## How a cluster runs **1. Pick the machines** Choose a GPU and how many machines (2 to 64, as many as the location has free), then one of the offers for that size. **2. Create** Every machine is reserved at once. Status is **provisioning** while they boot and join the private network. **3. Every machine is up** When every machine has joined, billing starts for all of them at the same moment. With software picked, status is **installing** while it installs; with **None** the cluster is **ready** straight away. **4. Ready** The software checks itself on every node (for Slurm, every node idle; for Kubernetes, every node Ready with its GPUs schedulable; for Ray, every node in `ray status`), then the cluster is **ready** and you get an email. **5. Terminate** Terminating stops every machine together, and billing for the whole cluster ends. ## All or nothing A cluster starts whole or not at all. If any machine fails, stops or disappears while the cluster starts, the cluster fails at once. Each step also has its own time limit, worked out from how long that kind of machine takes to boot: the machine running, then its cluster agent checking in, then every machine reaching the others. A machine that is late for a step fails the cluster then, and nothing waits past 25 minutes (longer on machines that are known to boot slowly). Runcrate releases every machine it reserved and the cluster is **failed**; its events say which node and which step. Nothing is billed for a cluster that never had every machine up, and nothing is started again without you: the cluster page shows the offers available now, and you choose. It ends whole too. One node of a cluster cannot be terminated on its own from the Instances page or the API: you terminate the cluster. When your balance runs out, or the environment reaches its [monthly budget](https://runcrate.ai/docs/instances/guardrails.md#monthly-budgets), the whole cluster is stopped. If the software fails to install after every machine is up, the cluster is **degraded**: the machines stay up and reachable over SSH, and you choose whether to retry the install or terminate. ## How many machines a location has Where a location says how many machines of a kind it has free, its offers sell up to that many (`nodes_available`, and `max_nodes` up to 64). Where it only says whether it has any, its offers stop at 8 machines (`nodes_available` is null and `max_nodes` is 8): one machine short fails the whole cluster, so bigger sizes are never started on a guess. Bigger clusters are reserved ahead: **Request a quote** under the node sizes. A size withheld this way reads *reserved ahead*; *not in stock* means the location counted fewer machines free. ## Billing * Billing starts when every machine has joined, from the same moment for every machine, and runs until the cluster is terminated. Nothing is billed while the machines are still starting. * Each machine is billed at its hourly price for the time it runs, the same way an instance is. The whole cluster costs the machine price times the number of machines, per hour. * A worker added to a running cluster is billed from the moment it is up. * Your balance must cover at least one hour of the whole cluster, and never less than \$10, to create one. * There are no prepaid or reserved terms on self-serve clusters. * On the **Billing** page a cluster is one line, **Cluster usage** and its name, which opens to a row per machine. Clusters have no maximum runtime or idle stop. Terminate the cluster when the work is done. ## How the machines are joined Self-serve clusters run on **Ethernet**: the machines' own network, with an encrypted private network on top (`10.77.0.0/24`). Some sites give their machines no public address of their own; there the private network runs over the site's network, and you reach each machine over SSH through a Runcrate gateway, the same as on any other cluster. The encrypted network moves about 1 to 2 Gbit/s between machines. That suits independent jobs on every node, Slurm job arrays, Ray tasks, Kubernetes workloads and LoRA fine-tuning. It is not built for training a full model across machines, where GPUs on different machines exchange gradients every step. Run the [NCCL test](https://runcrate.ai/docs/clusters/connecting.md#test-the-network) after creating a cluster to see the bandwidth you actually get. Clusters on a fast InfiniBand or RoCE network are coming; ask us for one with **Request a quote**. Each offer also names its kind of machine: virtual machine, bare metal or container. Kubernetes is offered on virtual machines and bare metal. ## Community hosts Offers are on datacenter hosts unless you ask for community hosts as well: **Include community hosts** in the dashboard, `include_community` in the API and MCP, or `--include-community` in the CLI. Community hosts are machines run by independent operators; keep secrets you would not give them off those clusters. A volume cannot be attached to a cluster on community hosts. ## Shared storage On virtual machines and bare metal every node mounts `/shared`, a folder exported from the head over the private network, with the hostfile in it. Attach a [volume](https://runcrate.ai/docs/storage/overview.md) when you create the cluster and its files are synced into `/shared/volume` on the head, and back to the volume, so they outlive the cluster. Container machines share no folder: each node keeps its own disk, and a volume shows on the head only. Everything else on the machines, `/shared` included, is deleted when the cluster is terminated. ## Request a quote For reserved capacity, very large clusters or a fast InfiniBand or RoCE network, use **Request a quote** on the **Clusters** page and the team gets back to you. Self-serve clusters and quoted clusters live side by side. ## Next steps > **[Create a cluster](https://runcrate.ai/docs/clusters/creating.md)** > > From the dashboard, your agent, the CLI or the API. > **[Software](https://runcrate.ai/docs/clusters/software.md)** > > Slurm, Kubernetes and Ray: what is installed and how to use it. > **[Connect and run](https://runcrate.ai/docs/clusters/connecting.md)** > > SSH, the hostfile, torchrun, mpirun, DeepSpeed and the NCCL test. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.