> ## Documentation Index
> Fetch the complete documentation index at: https://runcrate.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> IOMMU, reboots, unsupported or failing GPUs, offline machines and installs that stop.

Start with what the machine says about itself:

```bash wrap theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
# Registration, IOMMU, each GPU (and why it's blocked), VMs
sudo runcrate-host status

# The agent's log, live
sudo journalctl -u runcrate-host -f
```

The installer is safe to run again at any point: every step checks before it changes anything.

## Installing

> **IOMMU is off**
>
> The installer says `IOMMU is off. GPU passthrough needs it.` and shows the words it would add to the kernel command line, `GRUB_CMDLINE_LINUX_DEFAULT`:
>
> ```bash wrap theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
> # Intel
> intel_iommu=on iommu=pt
> # AMD
> amd_iommu=on iommu=pt
> ```
>
> Answer yes (or pass `--yes`) and it writes them to `90-runcrate-host.cfg` in `/etc/default/grub.d/`, runs `update-grub` and reboots. Run the same install command again when the machine is back.
>
> If it says `the kernel has 'intel_iommu=on iommu=pt' but no IOMMU came up` (or the AMD line), the kernel is ready but the firmware isn't: turn on VT-d (Intel) or AMD-Vi / IOMMU in the BIOS, reboot and run the command again.
>
> If it says `update-grub not found`, your machine boots without GRUB: add the same words to your boot loader's kernel command line yourself, reboot and run the command again.

> **Reboot pending**
>
> The IOMMU line only takes effect after a reboot, and nothing else is installed until then. If the installer added the line but the machine hasn't rebooted, reboot it. If you answered no, nothing was changed: run the command again and answer yes, or add the line yourself and reboot. After the reboot, run the same install command again. It asks for the registration token again, because the machine only registers once the IOMMU is on.
>
> Check that the IOMMU is on after the reboot: `sudo runcrate-host status` shows `IOMMU: true` once the agent is installed, and before that `ls /sys/kernel/iommu_groups` lists groups.

> **CPU virtualization is off**
>
> `CPU virtualization (VT-x / AMD-V) is off` or `/dev/kvm is missing`: turn on VT-x (Intel) or AMD-V (SVM) in the BIOS and run the command again.

> **The machine is refused**
>
> * `this is a virtual machine; Runcrate hosts must be bare metal`: install on the physical machine. GPU passthrough needs the real hardware.
> * `supported: Ubuntu 22.04, Ubuntu 24.04, Debian 12`: install one of those.
> * `no NVIDIA GPU found`: the machine shows no NVIDIA GPU on its PCI bus. Check the cards are seated and powered.
> * `these processes are using the GPUs`: stop the listed processes and run the command again. The installer won't take GPUs that are in use.

> **The token is refused**
>
> * `registration token expired` or `registration token used up`: a token works for 30 days, or until you revoke it under **Install tokens**. Create a new install command on [Hosts](https://runcrate.ai/dashboard/host).
> * `this host account has reached its host limit`: while Runcrate is still checking your company you can register 2 machines. The rest can join once the check is done.
> * `--token is gone`: the installer no longer takes the token on the command line. Leave it out to be asked for it, or use `--token-file` (see [Unattended installs](https://runcrate.ai/docs/hosts/install.md#unattended-installs)).
> * `--token-file ... must be owned by root and readable by root only`: run `chown root` and `chmod 600` on the file, as the message says.

> **The agent did not start**
>
> The installer prints the agent's last 30 log lines. The usual cause is the data directory: the agent only uses a directory that is new or empty (a fresh filesystem's `lost+found` is fine) and refuses one holding other files. If the data directory is a separate disk, the agent also refuses to start while it isn't mounted. Point `--data-dir` at an empty directory on fast local disk and run the command again.

## Installing or verifying for too long

The Hosts page follows a new machine through **Registered**, **Image checked**, **First heartbeat**, **GPU check** and **Online**. On a ready machine the install takes about 35 seconds before the GPU check starts.

> **Stuck before First heartbeat**
>
> The agent registered but hasn't reported yet. On the machine:
>
> ```bash wrap theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
> systemctl status runcrate-host
> sudo journalctl -u runcrate-host -f
> ```
>
> Look for errors reaching Runcrate (the machine needs outbound HTTPS), downloading or checking the VM image, or the data directory. Fix the cause and run the install command again.

> **Stuck on GPU check**
>
> The check boots a throwaway VM that tests every free GPU, the disk, the internet and the isolation. If it doesn't finish, the agent's log says why. A GPU that can't be passed through shows `BLOCKED` with the reason in `sudo runcrate-host status`:
>
> * `IOMMU is off`: see [IOMMU is off](#installing).
> * `IOMMU group ... also holds ...`: the GPU shares its IOMMU group with another device, so it can't be given to a VM alone. Turn on ACS in the BIOS or move the card to another slot.
> * `the kernel runs with pcie_acs_override`: remove `pcie_acs_override` from the kernel command line. It hides the real isolation between devices.
>
> Fix it, then choose **Re-run check** on the machine.

## The GPU check failed

The Hosts page shows why the check failed. A machine that fails its first check isn't sold until a check passes; on a machine already on sale, only the GPUs that failed come off sale.

> **GPU 2 is not a model Runcrate sells**
>
> The GPU isn't one Runcrate offers. Supported: NVIDIA datacenter cards (A100, H100, H200, B200, L40S, RTX 6000 and others) and GeForce RTX 30, 40 and 50 series. Checking again won't change that. On a machine already on sale, the other GPUs stay on sale. On one that never passed a check, [keep only the GPUs that passed](#keep-only-the-gpus-that-passed) to sell the others.

> **GPU 0 failed its memory test**
>
> The memory test writes over nearly all of the GPU's memory, and the matrix multiply test checks its results against the CPU. A failure usually means a faulty card, overheating or a power problem. Check the card's seating, cooling and power, then choose **Re-run check** on the machine. On a machine already on sale, choose **Drain** first and wait for its rentals to end before you power it off, and **Resume** once the check passes. If a machine that never passed keeps failing, [keep only the GPUs that passed](#keep-only-the-gpus-that-passed).

> **GPU 1 was not benchmarked**
>
> The check didn't get a result for that GPU. Look in `sudo runcrate-host status` for a `BLOCKED` reason and in the agent's log, fix it, then choose **Re-run check**.

## Keep only the GPUs that passed

On a machine already on sale you don't need this: the GPUs that passed stay on sale. `--gpus` only counts when a machine registers. A machine that is already registered keeps its GPUs, so installing again with `--gpus` changes nothing. Add the machine again as a new one instead. If anything is rented on it, empty it first: see [Before you uninstall](https://runcrate.ai/docs/hosts/uninstall.md#before-you-uninstall).

**1. Remove the machine**

On [Hosts](https://runcrate.ai/dashboard/host), choose **Remove machine** on the failed machine's card, or in its **Actions** menu under **Your hosts**. Its agent stops and logs `host credential rejected`; that is expected. **Add a machine** then opens filled in with its name, its site and the GPUs that passed, and the page keeps them until you make the token.

**2. Uninstall on the machine**

```bash wrap theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -fsSL https://get.runcrate.ai/host |
  sudo sh -s -- --uninstall
```

The new install can't register while the old agent is on the machine.

**3. Install it as a new machine**

Create the install command there and run it on the machine. It lists the GPUs that passed with `--gpus`; `lspci -D -d 10de:` lists their addresses on the machine. The uninstall removed the machine's settings, so if you installed it with `--data-dir`, add that to the line too.

```bash wrap theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -fsSL https://get.runcrate.ai/host |
  sudo sh -s -- \
  --name rack2-node-01 \
  --gpus '0000:41:00.0, 0000:43:00.0'
```

It runs its own GPU check, like any new machine.

## Offline

A machine that misses its reports for 60 seconds shows as **Offline**. Its listings aren't sold, and time a rental spends on it while it is offline is a [downtime refund](https://runcrate.ai/docs/hosts/payouts.md#downtime-refunds). While the agent isn't running, every VM's network is cut, so customers lose their network until it is back.

> **Bring an offline machine back**
>
> On the machine:
>
> ```bash wrap theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
> systemctl status runcrate-host
> sudo runcrate-host status
> sudo journalctl -u runcrate-host -f
> ```
>
> * **The agent isn't running:** `sudo systemctl start runcrate-host`. Systemd restarts it on its own after a crash.
> * **No internet:** the machine needs outbound HTTPS. Check its network, DNS and any proxy or firewall in front of it.
> * **Powered off or rebooted:** both services start at boot; the machine comes back by itself.

> **The agent stopped and won't restart**
>
> The agent stops for good, and stops every VM, when Runcrate no longer accepts the machine's credential: the machine was removed from the dashboard, or its credential was revoked. The log says `host credential rejected`. To use the machine again, [uninstall](https://runcrate.ai/docs/hosts/uninstall.md#uninstall-the-agent) and install it again with a new install command.

## Still stuck

> **[Email support](mailto:support@runcrate.ai?subject=Runcrate%20host%20help)**
>
> Send the output of `sudo runcrate-host status` and the last lines of the agent's log.

> **[What runs on your machine](https://runcrate.ai/docs/hosts/security.md)**
>
> Every file, service and rule the agent adds.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.