Run AI on your own hardware

A GPU can be cheaper than an API bill and still be the more expensive way to run your application. The difference is what happens after you buy it: how often it does useful work, whether its answers are good enough, and who fixes the service when it stops answering.

I would make the ownership decision around one workload. Take an internal document assistant that reads a passage and returns a short summary. It gives us something concrete to compare: the same documents, acceptable summaries, a response deadline, and a monthly bill. Once those are defined, the hardware discussion becomes much easier.

Put the bill next to the work

Here is a worked estimate, not a benchmark or a record of my electricity bill. Every price and workload value in this example is an assumption you can replace.

Suppose the assistant handles 2,000 requests a day, each with 2,000 input tokens and 500 output tokens. Tokens are the chunks of text the model processes and generates. Over 30 days, that is 120 million input tokens and 30 million output tokens.

At hypothetical API rates of $1 per million input tokens and $4 per million output tokens, the monthly cost is:

120 × $1 + 30 × $4 = $240

Now suppose the local machine costs $1,200, we spread that purchase over 24 months, and we assume an average draw of 200 watts across the whole day. At an assumed electricity rate of $0.15 per kWh:

That looks attractive beside $240. Reduce demand to 200 requests a day, though, and the same API assumptions produce a $24 bill. The local machine still occupies the desk and still needs maintenance. Its electricity use may fall, but the purchase doesn't become ten times cheaper.

This calculation also assumes that the local system delivers acceptable answers at the required speed. A cheaper summary that misses the decision in a document creates work for somebody else. Count rejected answers, retries, and human correction when comparing costs. If both systems need supervision, count the difference in that supervision.

There is no universal crossover at one million or ten million tokens a day. Input and output have different prices, tasks need different models, and a busy GPU has very different economics from one waiting for its next request. Use the rates for a hosted model that passes your task, including any caching or batch discounts you can actually use.

The machine has to finish the work

Our example asks the local system to generate one million output tokens a day. Spread evenly across 24 hours, that requires about 11.6 output tokens per second. Squeeze it into an eight-hour workday and the requirement rises to about 34.7. Neither number includes the time needed to process the input passages, and neither tells us how long a person waits during the busiest hour.

That is why I would test the workload before buying hardware. Define acceptance in terms a user would recognize: the summary preserves the main decision, doesn't invent a date, and arrives soon enough to be useful. Then run representative documents, including the long and awkward ones, at the expected concurrency. Record quality, time to the first token, total response time, and the number of requests completed.

A single stream's generation speed cannot establish the capacity of a shared service. Batching can increase total throughput while individual requests take longer. Likewise, a short prompt says little about a request carrying a large document. The capacity question is whether the complete workload fits inside the response deadline.

Memory buys room for the conversation

GPU memory is often the first constraint when choosing a local model. A model's parameters are the stored numbers it uses to generate an answer. At 16 bits per parameter, a nominal 32-billion-parameter model needs about 64 GB for those numbers alone. At an idealized four bits, the arithmetic drops to 16 GB. These are decimal GB calculations, before format overhead and runtime memory.

Quantization reduces the precision used to store the weights. Think of rounding measurements: you save space, but some distinctions disappear. Whether those distinctions matter depends on the task. A model that still chats comfortably may make more errors on the SQL transformations you care about.

For a concrete reference, Ollama lists Qwen2.5 32B in Q4_K_M at about 20 GB. That makes a 24 GB GPU worth investigating, but the download size is not a promise that every context length or concurrency setting will fit.

The model also needs working memory for the conversation. Its attention cache, often called the KV cache, stores information used while generating subsequent tokens. Longer contexts and more simultaneous requests consume more of that memory. Ollama's documentation explicitly connects context length and parallel requests to memory requirements.

The same weight-only calculation puts a nominal 7B model at 3.5 GB, 13B at 6.5 GB, 70B at 35 GB, and 405B at 202.5 GB using idealized four-bit storage. Add overhead and working memory before choosing a machine. Two GPUs also need an engine that can distribute the model effectively; their memory doesn't automatically become one shared pool.

Memory determines whether the workload fits. Compute and memory bandwidth then help determine how quickly it runs. CPU inference can be useful for small models or patient batch work, and Apple Silicon is another supported route. llama.cpp supports CPU, Metal, CUDA, and hybrid CPU/GPU execution. Measure the latency you need before deciding that a particular GPU is mandatory.

Choose the model and engine together

Start with the tasks: a code review, a SQL refactor, a tag classifier, or a document summary. They expose different weaknesses. A SQL refactor needs to preserve query behavior; a tag classifier needs to respect the allowed labels. Public benchmarks can help shortlist models, but your examples decide whether a candidate deserves the job.

Model size alone won't answer that question. Qwen2.5 32B is a concrete candidate for the memory calculation above. A smaller model might handle classification adequately and leave much more room for concurrent requests. A larger model might justify its cost for difficult summaries. Fine-tuning introduces another evaluation: whether the model's license, training tools, and serving engine support the adaptation you need.

The engine is the software that loads the model and serves requests. Its format support matters as much as the model name:

• llama.cpp is a natural starting point for GGUF files and local experiments. Qwen publishes a GGUF model card with llama.cpp instructions. Q4_K_M is a quantization variant used in that format, not a universal setting for every engine.
• Ollama simplifies downloading and managing local models. To download that candidate, use: ollama pull qwen2.5:32b. Downloading it is the beginning of the capacity test.
• vLLM is worth evaluating for a shared service. Its batching and attention-memory management support serving concurrent requests. Match the exact model and quantization implementation to your hardware. AWQ and GPTQ are quantization approaches, while GGUF is a file format; they aren't interchangeable labels.
• Text Generation Inference, from Hugging Face, is now in maintenance mode. That changes how I would assess it for a new deployment.

In particular, don't assume that a Q4_K_M file will give the same experience across engines. vLLM's GGUF documentation describes its support as experimental and under-optimized, with a separate plugin required. A model that loads successfully still needs a quality and performance test in the exact configuration you intend to operate.

Owning the service means owning the boundaries

A home-lab deployment can place an inference service behind a reverse proxy such as Traefik or Caddy, with applications talking to a local HTTP endpoint. Proxmox, a dedicated LXC container, and Docker are possible layers, but each adds a boundary to configure. LXC shares the host kernel; exposing GPU devices there differs from assigning a PCI device to a virtual machine.

For Docker, --gpus all depends on a working host driver and the NVIDIA container integration. Follow the NVIDIA Container Toolkit setup for the runtime you use. Treat driver and engine upgrades as changes to test, with a known working configuration available for rollback.

An OpenAI-compatible endpoint such as http://your-host:8000/v1 can reduce client changes, but compatibility covers specific APIs and features. Check the model name, authentication, chat template, and requested features against the vLLM server documentation. Changing the base URL doesn't establish that tool calls or structured outputs behave identically.

Keeping inference local lets you control where prompts are processed. That control extends beyond the model process: logs, backups, monitoring, external tools, and network access all affect where the data goes. A reverse proxy with TLS protects transport; it still needs an access policy. Local hardware gives you the ability to define those boundaries and the responsibility to maintain them.

I would keep request identifiers, token counts, errors, and latency in a database such as Postgres. Store prompt and response text only where the debugging need justifies the access and retention policy. Otherwise, the private document assistant can quietly become a second private document archive.

vLLM exposes Prometheus metrics at /metrics, including request timing and queue information. Combine those with GPU monitoring. A small generation probe, for example a five-token request every 30 seconds, can detect a stalled service, while representative task checks tell you whether it still performs useful work. A healthy process and a useful answer are separate things to monitor.

When control is worth the work

Self-hosting is attractive when a suitable local model meets the task's quality bar, the workload keeps the machine usefully occupied, and someone can operate it within the response and availability requirements. It can also be worth paying more to keep processing inside a boundary you control or to choose when a model changes.

That stability requires more than saving the weights. Keep the tokenizer, prompt template, engine version, and generation settings with the deployment. Then test upgrades against the same representative tasks. You control the change schedule; you still need to check what each change does.

In the worked example, local ownership has room to save money at 2,000 requests a day and loses its cost advantage at 200. The purchase becomes defensible only after a load test confirms that the local model can handle those requests acceptably. Start there: compare completed, useful work at a known operating cost. The resulting answer will tell you whether to buy a GPU, keep using an API, or reserve each for the work it handles best.