> ## Documentation Index
> Fetch the complete documentation index at: https://runcrate.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Connect and run

> SSH to the head and the nodes, the hostfile, torchrun, mpirun and DeepSpeed across a cluster, /shared, and the NCCL bandwidth test.

The cluster page shows every command on this page with your cluster's own addresses filled in, and `GET /clusters/{id}` returns them under `connection`. Addresses and SSH commands appear once every machine is up and billed.

## SSH

Node 0 is the head. Each node is also an instance of its own, so the web terminal and VS Code work on every node from the cluster page, and the instance tools (`ssh_execute`, `file_upload`, `file_download`, `/instances/{id}/exec`) take a node's `instance_id`.

```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
runcrate clusters ssh train                 # the head
runcrate clusters ssh train --node 2        # node 2
runcrate clusters ssh train -- nvidia-smi   # one command on the head
```

The CLI signs a short-lived SSH certificate for the node each time, so no key is needed. If you added your own SSH key when creating the cluster, each node's `ssh_command` works with it too. On a machine with no public address of its own, `ssh_command` goes through a Runcrate gateway, with its own host and port; everything else works the same.

Between the nodes, root can `ssh node-1` without a password. That key is only accepted from the cluster's private network.

## The hostfile and launcher settings

`/shared/hostfile` has one line per node with its GPUs:

```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
node-0 slots=8
node-1 slots=8
node-2 slots=8
node-3 slots=8
```

Container machines share no folder, so there the hostfile is at `/etc/runcrate/hostfile` on every node.

Every login shell has `MASTER_ADDR` (the head on the private network), `MASTER_PORT` (29500), `NODE_RANK`, `WORLD_NODES`, `GPUS_PER_NODE`, and `NCCL_SOCKET_IFNAME` and `GLOO_SOCKET_IFNAME` set to the private network, from `/etc/runcrate/cluster.env`.

## torchrun

Run the same command on every node. From the head, this starts it on each node over the cluster's own SSH. Commands over SSH don't start a login shell, so each loads its settings first; the rendezvous gives every node its rank:

```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
for n in $(cut -d' ' -f1 /shared/hostfile); do
  ssh "$n" 'set -a; . /etc/runcrate/cluster.env; set +a; unset RANK; torchrun \
    --nnodes=$WORLD_NODES --nproc-per-node=$GPUS_PER_NODE \
    --rdzv-backend=c10d --rdzv-endpoint=$MASTER_ADDR:$MASTER_PORT --rdzv-id=train \
    /shared/train.py' &
done; wait
```

## mpirun

From the head, one process per GPU across the hostfile:

```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
mpirun --allow-run-as-root --hostfile /shared/hostfile -np 32 \
  -x NCCL_SOCKET_IFNAME -x MASTER_ADDR -x MASTER_PORT \
  python /shared/train.py
```

## DeepSpeed

DeepSpeed reads the same hostfile and starts every node over SSH from the head:

```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
cd /shared
deepspeed --hostfile /shared/hostfile \
  --master_addr $MASTER_ADDR --master_port $MASTER_PORT \
  train.py --deepspeed --deepspeed_config ds_config.json
```

With Slurm, Kubernetes or Ray, see [Cluster software](https://runcrate.ai/docs/clusters/software.md) for `sbatch`, `kubectl` and `ray job submit`.

## Shared files

* On virtual machines and bare metal, `/shared` is on the head and mounted on every node over the private network. Put code, configs and checkpoints your nodes share there; keep large datasets on each node's local disk when you can.
* Every node can read and write everything in `/shared`, whichever node made it. Files written from the other nodes show as owned by `nobody`, which changes nothing for root.
* With a volume attached, its files are in `/shared/volume` and are synced back to the volume, so they outlive the cluster.
* Container machines share no folder: each node keeps its own disk, a volume shows on the head only, and you copy files between nodes over SSH.
* Everything on the machines, `/shared` included, is deleted when the cluster is terminated.

## Test the network

The NCCL test runs an all-reduce across every GPU on every node, from 1 MB to 1 GB messages, and reports the bus bandwidth for each size and the peak. It takes a few minutes and stops after 10. One test runs at a time per cluster, and it uses every GPU, so run it before your job. On a slow link it stops after 5 sizes and says so, since the larger sizes would take many minutes of billed time.

* Dashboard: **Run NCCL test** under **Network test** on the cluster page.
* Agent: `run_cluster_nccl_test`, then `get_cluster` for `last_test`.
* CLI: `runcrate clusters test train`.
* API: `POST /clusters/{id}/tests`, then `GET /clusters/{id}/tests/{testId}`.

Right after a cluster becomes ready the test tools may still be preparing in the background. If the test says so, try again a few minutes later.

## Tunnels

Kubernetes and Ray are never open to the internet. `runcrate clusters tunnel train` forwards them to your computer over SSH to the head:

| Software | Forward | Then |
| - | - | - |
| Kubernetes | `6443`, to the head's address on the private network | `kubectl` with the cluster's kubeconfig |
| Ray | `8265`, to the head's own loopback | the Ray dashboard at `http://127.0.0.1:8265`, and `ray job submit` |

The cluster's `connection.snippets` has both tunnels as plain `ssh -N -L` commands (`kubectl_tunnel`, `ray_tunnel`).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.