> ## Documentation Index
> Fetch the complete documentation index at: https://runcrate.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Cluster software

> What None, Slurm, Kubernetes and Ray install on a cluster, when each is ready, and how to use it.

Every cluster, whatever its software, has the same base on every node:

* the private network, with every node reachable as `node-0`, `node-1`, ... (node 0 is the head);
* SSH between the nodes as root, so `mpirun` and `pdsh` work with `node-*` names;
* `/shared`, mounted from the head, with the hostfile at `/shared/hostfile` (container machines share no folder and keep the hostfile at `/etc/runcrate/hostfile`);
* the launcher settings in `/etc/runcrate/cluster.env`, exported into every login shell: `NODE_RANK`, `WORLD_NODES`, `GPUS_PER_NODE`, `MASTER_ADDR` (the head on the private network), `MASTER_PORT` (29500), `NCCL_SOCKET_IFNAME` and `GLOO_SOCKET_IFNAME`.

A script that runs outside a login shell can load them with `set -a; . /etc/runcrate/cluster.env; set +a`.

The software you pick installs on top once every node is up. The cluster is **installing** meanwhile and **ready** once the software has checked itself on every node. If the install fails the cluster is **degraded**, with the reason in `stack_detail`: the nodes stay up, and you can retry the install or terminate.

## None

Nothing more is installed. The cluster is ready as soon as every node has joined. Run your own launcher with the hostfile and the settings above: see [Connect and run](https://runcrate.ai/docs/clusters/connecting.md).

## Slurm

Slurm with munge authentication. `slurmctld` runs on the head and `slurmd` on every node, the head included, so the head's GPUs take work too.

* **Partition:** `gpu`, the default, with every node in it (`node-0`, `node-1`, ...).
* **GPUs** are generic resources: ask for them with `--gres=gpu:N` or `--gpus-per-node=N`.
* **Files:** `/home` is local to each node; put code and data your jobs share in `/shared`.
* **Ready** when `sinfo` shows every node idle.
* There is no accounting database, so `sacct` is not available. `squeue` and `sinfo` are, and the cluster page shows the live queue.

A batch job across every node, from the head:

```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
cat > /shared/train.sbatch <<'EOF'
#!/bin/bash
#SBATCH --job-name=train
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --output=/shared/logs/%x-%j.out

set -a; . /etc/runcrate/cluster.env; set +a
srun torchrun \
  --nnodes "$SLURM_NNODES" --nproc-per-node "$GPUS_PER_NODE" \
  --rdzv-backend c10d --rdzv-endpoint "$MASTER_ADDR:$MASTER_PORT" \
  /shared/train.py
EOF
mkdir -p /shared/logs
sbatch /shared/train.sbatch
squeue
```

## Kubernetes

A lightweight Kubernetes (k3s) cluster: the control plane on the head and every node an agent, on the private network. The NVIDIA container toolkit and device plugin are installed, so pods can ask for GPUs.

* **Ready** when every node is `Ready` and has `nvidia.com/gpu` allocatable.
* **On the head:** `kubectl` works as root.
* **From your computer:** download the kubeconfig and open the tunnel; its server is `https://127.0.0.1:6443`, reached through SSH to the head, so the API is never open to the internet.
* The built-in ingress controller is off.
* Offered on virtual machines and bare metal, not on container clusters.

```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
runcrate clusters kubeconfig train -o kubeconfig.yaml
runcrate clusters tunnel train          # keep it running, then in another terminal:
export KUBECONFIG=$PWD/kubeconfig.yaml
kubectl get nodes
```

The tunnel forwards your `127.0.0.1:6443` to the API server on the head's address on the private network, as `ssh -N -L 6443:10.77.0.1:6443 root@<head address>`; the API returns the cluster's own command in the `X-Runcrate-Tunnel` header of `GET /clusters/{id}/kubeconfig`. The kubeconfig grants admin on the cluster: keep it private.

A GPU pod uses the `nvidia` runtime class and a GPU limit:

```yaml theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
apiVersion: v1
kind: Pod
metadata:
  name: gpu-check
spec:
  runtimeClassName: nvidia
  restartPolicy: Never
  containers:
    - name: cuda
      image: nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 8
```

## Ray

Ray 2.59 on Python 3.12, in its own environment at `/opt/runcrate/ray`. The head runs `ray start --head` on the private network (port 6379) and every other node joins it.

* **Ready** when `ray status` shows every node.
* `ray` is on the path on every node. To import Ray in your own code use `/opt/runcrate/ray/bin/python`, or `source /opt/runcrate/ray/bin/activate` first.
* The **dashboard** listens on the head at `127.0.0.1:8265`. Open it from your computer with `runcrate clusters tunnel train`, then browse to `http://127.0.0.1:8265`; jobs submit through the same address.

```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
runcrate clusters ssh train -- ray status

# From your computer, with the tunnel running
RAY_ADDRESS=http://127.0.0.1:8265 ray job submit --working-dir . -- python train.py
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.