latent
Most of what I run against LLMs day to day goes to hosted APIs, and for good reason — nobody's home GPU is competing with a datacenter. But there's a class of work where local wins: anything touching data that shouldn't leave the building, anything you want to hammer in a loop without watching a billing meter, and anything built on a model you've fine-tuned yourself. For that class, on a single consumer NVIDIA GPU, Ollama is the tool I reach for. Not because it's the fastest serving stack — it isn't — but because it makes the operational side boring, and boring is what you want from infrastructure.
The catch is that Ollama's defaults are tuned for the demo experience: type ollama run, get tokens, feel good. Running it as a piece of infrastructure your applications actually depend on takes a handful of deliberate choices. Here's where I'd spend the attention.
Pick the quantization like you pick an index
Every model on the Ollama registry comes in multiple quantizations, and the tag you pull matters more than almost anything else you'll configure. The tradeoff is the same shape as any storage decision: smaller types buy you capacity and speed at the cost of precision, and the question is whether the precision you gave up was doing anything useful.
For most work, 4-bit is the right call — the q4_K_M flavor of GGUF has become the community default because it sits at the point where quality loss is hard to notice in normal use but the memory savings are enormous. As a rule of thumb, a 4-bit model needs roughly half a gigabyte of VRAM per billion parameters for the weights, so an 8B model fits comfortably on a mid-range card and a 30B-class model becomes plausible on a 24 GB one. Jumping to 8-bit roughly doubles that footprint for a quality bump you'll struggle to measure on most tasks; dropping below 4-bit starts to show up as real degradation, especially in reasoning and code. My rule: q4_K_M by default, q8_0 only when I've seen an actual failure I can attribute to quantization, which is rare.
The part people miss is that weights aren't the whole bill. The KV cache — the memory that holds your context window — grows with context length, and on long contexts it can rival the weights themselves. Which brings us to the settings.
The three settings that matter
First, context length. Ollama ships with a small default context — a few thousand tokens — and it will happily, silently truncate everything beyond it. If you're doing RAG or agent work, that silence is poison: your retrieved passages fall out of the window and the model answers from vibes. Set num_ctx deliberately, either per model in a Modelfile or per request through the API. But set it with the VRAM bill in mind, because a large context on a large model can push the KV cache past what your card holds, and Ollama will start splitting the model between GPU and CPU. ollama ps tells you the split. If you see anything other than 100% GPU, your tokens-per-second just fell off a cliff and you should either shrink the context or the model.
Second, keep_alive. By default a model unloads five minutes after the last request, and loading several gigabytes of weights back into VRAM is not instant. For an interactive app, a cold start in the middle of a session reads as "the site is broken." I set OLLAMA_KEEP_ALIVE long — or to -1 to pin the model — on any box whose job is serving one model. The RAM is already paid for; use it.
Third, concurrency. OLLAMA_NUM_PARALLEL controls how many requests a loaded model serves at once, and OLLAMA_MAX_LOADED_MODELS controls how many models share the card. Both default low, and on a single GPU that's correct — every parallel slot multiplies the KV cache, and every extra loaded model is another set of weights resident. Resist the urge to crank them. If you genuinely need high concurrent throughput, you've outgrown the tool: that's vLLM territory, with continuous batching and PagedAttention, and it earns its extra operational weight at that scale. One GPU, a handful of users, mixed models — Ollama. Many users, one model, throughput as a KPI — vLLM.
Run the server under systemd or Docker, doesn't matter much — but treat the environment variables as configuration under version control, not something you exported in a shell once and forgot. Six months later, "why does staging behave differently" is always an environment variable.
Your own fine-tune, served like any other model
This is the part that justifies the whole local setup, because hosted APIs serve their models, not yours. Ollama will serve anything you can get into GGUF, which means a fine-tune you trained yourself slots in next to the off-the-shelf ones behind the same API.
The path depends on what your training run produced. If you did a full fine-tune and have safetensors weights, you convert them with llama.cpp's conversion script, quantize the result down to your serving precision, and wrap it in a Modelfile. If you did a LoRA — which on a single consumer GPU is almost certainly what you did, since parameter-efficient training is the only kind that fits — you keep the adapter separate and let the Modelfile stack it on the base:
FROM llama3.1:8b
ADAPTER ./my-lora-adapter.gguf
PARAMETER num_ctx 8192
PARAMETER temperature 0.3
SYSTEM You classify support tickets. Answer with the category only.
Then ollama create tickets -f Modelfile and it's a first-class model: ollama run tickets, same REST API, same lifecycle as everything else. The Modelfile is the piece I'd praise most in the whole system — it's a Dockerfile for model configuration. Base weights, adapter, decoding parameters, chat template, system prompt, all in one small text file you can diff and check into git. The alternative — scattering those decisions across application code, where every caller sets its own temperature — is how you get behavior nobody can reproduce.
One warning from experience with the format wars: make sure the chat template in your Modelfile matches what the model was fine-tuned with. A mismatched template is the classic silent failure — no error, no crash, just a model that seems oddly dumber than it was in your training evals, because the prompt tokens are framed differently than anything it saw in training.
And before you fine-tune at all, be honest about whether you need to. A fine-tune bakes in behavior — a format, a tone, a narrow classification skill. It does not reliably add knowledge, and it goes stale the moment your data changes. If the problem is "the model doesn't know my documents," that's retrieval, not training; I've written before about that split and I keep landing in the same place. The fine-tunes that have been worth it for me are the small, boring ones: strict output formats and narrow classifiers, where a tuned 8B beats a prompted model three times its size and runs on hardware I own.
Wire it in through the OpenAI-compatible endpoint
Ollama exposes an OpenAI-compatible API at /v1 alongside its native one, and that's the integration point I'd standardize on. Point the OpenAI SDK at http://localhost:11434/v1, set the model name, done — any codebase already written against the hosted API can now talk to your GPU with a one-line base-URL change. That makes local-versus-hosted a routing decision instead of a rewrite: cheap bulk work and private data go to the local model, the hard reasoning goes to a frontier model, and the application code can't tell the difference.
That's the whole pitch, really. Ollama's job is to make a local model — including one carrying your own adapter — look like just another API endpoint, on hardware you control, with a config file you can read. Get the quantization right, size the context against your VRAM honestly, pin the model in memory, and put the Modelfile in git. None of it is glamorous. It's the same discipline as running any other stateful service, applied to a stack that most people still treat as a toy.