Cluster software

• the private network, with every node reachable as node-0, node-1, … (node 0 is the head);
• SSH between the nodes as root, so mpirun and pdsh work with node-* names;
• /shared, mounted from the head, with the hostfile at /shared/hostfile (container machines share no folder and keep the hostfile at /etc/runcrate/hostfile);
• the launcher settings in /etc/runcrate/cluster.env, exported into every login shell: NODE_RANK, WORLD_NODES, GPUS_PER_NODE, MASTER_ADDR (the head on the private network), MASTER_PORT (29500), NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME.

​

None

​

Slurm

• Partition: gpu, the default, with every node in it (node-0, node-1, …).
• GPUs are generic resources: ask for them with --gres=gpu:N or --gpus-per-node=N.
• Files: /home is local to each node; put code and data your jobs share in /shared.
• Ready when sinfo shows every node idle.
• There is no accounting database, so sacct is not available. squeue and sinfo are, and the cluster page shows the live queue.

cat > /shared/train.sbatch <<'EOF'
#!/bin/bash
#SBATCH --job-name=train
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --output=/shared/logs/%x-%j.out

set -a; . /etc/runcrate/cluster.env; set +a
srun torchrun \
  --nnodes "$SLURM_NNODES" --nproc-per-node "$GPUS_PER_NODE" \
  --rdzv-backend c10d --rdzv-endpoint "$MASTER_ADDR:$MASTER_PORT" \
  /shared/train.py
EOF
mkdir -p /shared/logs
sbatch /shared/train.sbatch
squeue

​

Kubernetes

• Ready when every node is Ready and has nvidia.com/gpu allocatable.
• On the head: kubectl works as root.
• From your computer: download the kubeconfig and open the tunnel; its server is https://127.0.0.1:6443, reached through SSH to the head, so the API is never open to the internet.
• The built-in ingress controller is off.
• Offered on virtual machines and bare metal, not on container clusters.

runcrate clusters kubeconfig train -o kubeconfig.yaml
runcrate clusters tunnel train          # keep it running, then in another terminal:
export KUBECONFIG=$PWD/kubeconfig.yaml
kubectl get nodes

apiVersion: v1
kind: Pod
metadata:
  name: gpu-check
spec:
  runtimeClassName: nvidia
  restartPolicy: Never
  containers:
    - name: cuda
      image: nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 8

​

Ray

• Ready when ray status shows every node.
• ray is on the path on every node. To import Ray in your own code use /opt/runcrate/ray/bin/python, or source /opt/runcrate/ray/bin/activate first.
• The dashboard listens on the head at 127.0.0.1:8265. Open it from your computer with runcrate clusters tunnel train, then browse to http://127.0.0.1:8265; jobs submit through the same address.

runcrate clusters ssh train -- ray status

# From your computer, with the tunnel running
RAY_ADDRESS=http://127.0.0.1:8265 ray job submit --working-dir . -- python train.py