Skip to main content
Run Ollama on a dedicated cloud GPU instead of your local machine. Faster inference, larger models, and a shared endpoint for your team.

1. Deploy and install

2. Start the server

3. Pull models

4. Test the API

5. Use the OpenAI-compatible endpoint

6. Larger models on A100 80 GB

For 70B models, use an A100 80 GB (llama3.1:70b is about 43 GB at its default 4-bit quantization, more than a 40 GB A100 holds):

Monitoring

Tips

  • Ollama quantizes models by default (Q4). For higher quality, use :fp16 tags if VRAM allows.
  • The first request after pulling a model is slower: Ollama loads into GPU memory on demand.
  • For production workloads with high concurrency, use vLLM instead.

Cleanup