1. Prepare your input file
Create a JSONL file with one prompt per line:2. Deploy and upload
3. Upload the batch script
vLLM is installed on the Inference template. It finishes installing a few minutes after boot;runcrate ssh batch-job -- "systemctl start runcrate-python" returns once it has.
4. Download results and tear down
Monitor progress
Time estimate
Multiply the time by the GPU’s rate on the live GPU pricing page to estimate cost.
Tips
- vLLM offline batch mode uses continuous batching: much faster than sequential API calls.
- For 100K+ prompts, split the file and process in chunks to avoid OOM.
- Check your balance before starting:
runcrate billing balance.