What you’ll build
A self-hosted inference API that serves Qwen 2.5 7B (or any model that fits the GPU) on an A100, reachable at the instance’s public IP. You can point your existing OpenAI SDK code at it. Pick the GPU by the model’s weights plus room for the KV cache: a 7B or 8B model at BF16 needs about 16 GB, so one A100 (40 GB) is plenty. A 70B model at BF16 needs about 140 GB, so it needs several GPUs (see Multi-GPU serving). Gated models such as Meta’s Llama also need a Hugging Face token on the instance (HF_TOKEN).
Why vLLM
vLLM uses PagedAttention to manage GPU memory efficiently: on an 80GB H100 running a 7B FP16 model, this means serving 100+ concurrent requests instead of ~30. The V1 engine (default since v0.6.0) added disaggregated prefill/decode, preventing long prompts from blocking in-flight requests.Option A: CLI
1. Deploy the instance
2. Start the server
vLLM is installed on the Inference template (ubuntu-inference), built for the machine’s CUDA. It finishes installing a minute or so after the instance is up; the first command returns once it has.
3. Test it
4. Point your app at it
Option B: Python SDK
Option C: MCP (via Claude Code / Cursor)
“Deploy an A100 instance called llm-server with the Inference template. Once it’s ready, start serving Qwen 2.5 7B on port 8000. Give me the IP when it’s up.”Your AI assistant will:
- Call
create_instancewithname: "llm-server",gpu: "A100"andtemplate: "ubuntu-inference" - Poll
instance_statusuntil deployed - Call
ssh_executeto wait for vLLM (systemctl start runcrate-python) and start the server - Return the IP from
get_instance