Skip to main content
Serve any open-source LLM behind an OpenAI-compatible endpoint on your own GPU. This guide uses vLLM, the production standard for LLM serving: the same engine Stripe uses to process 50M+ daily API calls.

What you’ll build

A self-hosted inference API that serves Qwen 2.5 7B (or any model that fits the GPU) on an A100, reachable at the instance’s public IP. You can point your existing OpenAI SDK code at it. Pick the GPU by the model’s weights plus room for the KV cache: a 7B or 8B model at BF16 needs about 16 GB, so one A100 (40 GB) is plenty. A 70B model at BF16 needs about 140 GB, so it needs several GPUs (see Multi-GPU serving). Gated models such as Meta’s Llama also need a Hugging Face token on the instance (HF_TOKEN).

Why vLLM

vLLM uses PagedAttention to manage GPU memory efficiently: on an 80GB H100 running a 7B FP16 model, this means serving 100+ concurrent requests instead of ~30. The V1 engine (default since v0.6.0) added disaggregated prefill/decode, preventing long prompts from blocking in-flight requests.

Option A: CLI

1. Deploy the instance

Wait for it to deploy:

2. Start the server

vLLM is installed on the Inference template (ubuntu-inference), built for the machine’s CUDA. It finishes installing a minute or so after the instance is up; the first command returns once it has.

3. Test it

4. Point your app at it


Option B: Python SDK


Option C: MCP (via Claude Code / Cursor)

“Deploy an A100 instance called llm-server with the Inference template. Once it’s ready, start serving Qwen 2.5 7B on port 8000. Give me the IP when it’s up.”
Your AI assistant will:
  1. Call create_instance with name: "llm-server", gpu: "A100" and template: "ubuntu-inference"
  2. Poll instance_status until deployed
  3. Call ssh_execute to wait for vLLM (systemctl start runcrate-python) and start the server
  4. Return the IP from get_instance

Multi-GPU serving

For larger models, split the weights across GPUs with tensor parallelism. A 72B model at BF16 is about 145 GB of weights, so four H100s (320 GB) leave room for the KV cache:

Monitoring

Cleanup