Self-Hosting AI: Running LLMs on Your Own Hardware

The cloud made AI accessible. Self-hosting makes it yours.

I've been running models locally for a while now, partly out of curiosity, partly because I got tired of watching API bills creep up every month. The honest answer is that for a lot of workloads, self-hosting now makes sense. Not for everything. But more than people assume.

Why bother

Three reasons people end up here.

The first is cost at scale. If you're making a few hundred API calls a day, the cloud wins, no contest. If you're making a few hundred thousand, the math shifts. A single 24GB consumer GPU can run a quantized 32B model at decent throughput. Amortized over a year, that's cheaper than the same volume on a frontier API — and the gap widens fast as your traffic grows.

The second is data privacy. There are workloads I don't want to send to a third party, full stop. Customer documents, internal repos, anything covered by an NDA. Running the model on hardware I control closes that question. The compliance story is also shorter when nothing leaves the network.

The third is independence from API drift. Vendors deprecate endpoints. They change pricing. They tighten rate limits. They sometimes silently swap in a smaller model behind the same name. When you control the weights, the model behaves the same on Tuesday as it did on Monday.

The hardware bottleneck is VRAM

Forget compute, forget cores, forget bandwidth for a minute. The question that determines what you can run is: how much GPU memory do you have?

Rough rules of thumb at 4-bit quantization:

7B model → fits in 6GB
13B model → fits in 10GB
32B model → fits in 20GB
70B model → fits in 40GB (needs 2x consumer GPUs or one A6000)
405B model → needs a serious cluster

The 24GB cards (RTX 3090, 4090, A5000) are the sweet spot for serious work right now. They run a quantized 32B comfortably with room for context. A used 3090 from 2024 still does this job fine. You don't need an H100.

CPU inference exists. It works. It's slow enough that I only use it for batch jobs where latency doesn't matter. For interactive use, GPU is the only honest answer.

The home-lab approach

I run mine on a single Proxmox host with GPU passthrough into a dedicated LXC container. Docker on top of that. The container exposes an OpenAI-compatible HTTP endpoint, and everything else on my network just talks to that.

The pieces that matter:

Proxmox — bare-metal hypervisor, free, handles the GPU passthrough cleanly.
NVIDIA driver + CUDA toolkit in the container, matching the host version.
Docker with GPU runtime — --gpus all and you're done.
A reverse proxy — I use Traefik for TLS, but Caddy works the same.

Setting this up the first time took me an afternoon. The second time, twenty minutes. The hard part is GPU passthrough, and the documentation for that has gotten dramatically better in the last year.

Picking a model

There are too many models. Here's how I narrow it down.

If I want general chat and coding help, Llama 3.3 70B (or its successors) is the default. If I'm space-constrained, Qwen 2.5 32B punches well above its size, especially on multilingual and reasoning tasks. For very small machines, Phi-4 or a Llama 3.2 3B handles structured output and simple instruction-following surprisingly well.

For anything I want to fine-tune, I stick to Mistral or Llama variants because the tooling is mature.

The benchmark I actually care about is "does it work for my prompt." I keep a folder of representative tasks — a code review, a SQL refactor, a tag classifier, a long-form summarization — and I run them against any new model before promoting it. Public benchmarks lie often enough that I don't trust them without a sanity check.

Inference frameworks

You don't run the raw weights. You run them through an inference engine. The four that matter:

vLLM. What I default to for production. Implements PagedAttention, batches requests efficiently, handles continuous batching out of the box. OpenAI-compatible API mode is a one-flag thing. It's the closest thing to "industrial-grade" in the open-source world.

llama.cpp. Pure C++, runs on anything, including CPUs and Apple Silicon. Quantization-friendly (GGUF format). I use it on my laptop when I'm on the road. For server use, vLLM is faster.

Ollama. A wrapper around llama.cpp with a much nicer model-management UX. ollama pull qwen2.5:32b and you're done. Great for development, fine for low-traffic production.

TGI (Text Generation Inference). Hugging Face's offering. Solid, but I don't see it picked over vLLM much anymore.

I run vLLM in production and Ollama on my laptop. That covers everything I need.

Quantization in practice

Quantization is how you fit a 32B model into 24GB. The full-precision weights would need 64GB. We compress them to 4 bits per weight and trade a small amount of quality for a 4x memory reduction.

The formats you'll see:

GGUF — llama.cpp's format. Lots of tooling, very portable.
AWQ — activation-aware. Better quality at the same bit-rate, slightly slower.
GPTQ — older but well-tested. Most models on Hugging Face have a GPTQ variant.
Q4_K_M — the GGUF quantization level I default to. Quality is indistinguishable from full precision for chat-style workloads.

I haven't found a workload where quality dropped enough to matter when going from FP16 to Q4_K_M. The exception is heavy reasoning tasks, where Q5 or Q6 buys you back a measurable amount of accuracy.

Serving and ops

Once vLLM is up, you have an OpenAI-compatible endpoint at http://your-host:8000/v1. Any client library that talks to OpenAI talks to it. The only thing you change is the base URL and the API key.

What I add on top:

A request log. Every prompt, every response, latency, token count. Postgres works fine. It's the cheapest debugging tool you can install.
Prometheus metrics. vLLM exposes them at /metrics. Track tokens-per-second, time-to-first-token, GPU utilization, queue depth. When latency degrades you want to know which lever moved.
A health check. A 5-token "ping" prompt every 30 seconds. If the model OOMs or the GPU fault-resets, you find out immediately.

For batching, vLLM does it for you — continuous batching means the engine fills the GPU even with bursty traffic. You don't need to engineer this yourself.

When self-hosting actually beats the API

The crossover is closer than people think. Roughly:

At 1M tokens/day on a frontier model, the cloud is cheaper.
At 10M tokens/day on a mid-tier model, self-hosted on a 3090 wins.
At 100M tokens/day, self-hosted is dramatically cheaper, and you'll outgrow a single GPU.

These numbers slide based on your power costs, your hardware capex, and which API you're comparing to. The cloud is winning the very-low-volume game and losing the very-high-volume game. The middle is genuinely a judgment call.

The operational burden

I'd be lying if I said this was free. You inherit problems the API hides from you.

GPUs fail. Power supplies fail. Driver updates break CUDA compatibility. Models get superseded and you have to decide whether to upgrade. New inference engines come out and you have to decide whether to migrate. Quantization formats evolve. Context windows grow and you have to retune your serving config.

For a small team, that adds up to maybe a half-day a week of ops work, on average. For a single hobbyist running a personal model, it's closer to an hour a month after the initial setup.

The work is real. So is the control you get in exchange. I'd rather debug my own GPU than wait for an API to un-break itself.

What I actually run

For the curious: a single 3090 in a Proxmox LXC, vLLM serving Qwen 2.5 32B at Q4_K_M, behind Traefik with TLS. About 35 tokens/second on average, 60 on simple prompts, 20 on long-context. Total electricity cost: roughly $15/month at my rate. The hardware paid for itself in under five months at my volume. Your numbers will differ.

The cloud is fine. Self-hosting is also fine. Pick the one that matches what you actually need.