Every hardware guide I've written — the Mac mini build, the buying guide, the DGX-versus-AMD showdown — ends at the same unsatisfying place: you now own a very capable box. None of them tell you what to put on it.
This is that guide. Four runtimes worth knowing, a quantization rule that takes two minutes to learn and saves you a week of disappointment, and an honest answer to "which models are actually worth running locally right now." Everything here was verified against current releases in September 2026, because this corner of the world rewrites itself roughly every quarter.
The Two Numbers That Explain Everything
Before any software: local LLM speed is two separate measurements, and confusing them is why people buy the wrong machine and the wrong software.
Prefill is how fast the machine reads your input. It's compute-bound. You feel it when you paste a long document and wait. Decode is how fast the answer streams back. It's memory-bandwidth-bound. You feel it every single turn of every conversation.
Almost every runtime choice below is a trade between these two. Almost every benchmark you'll see quoted online reports only one of them without saying which.
The Four Runtimes
Ollama — start here
Ollama is the default answer for most people, and the reason is ergonomics: ollama run qwen3.8 and you're talking to a model. It ships a desktop app for macOS and Windows now (Linux stays CLI), keeps a curated model registry that feels like Docker Hub, and — the part that matters for building things — it runs an always-on server on port 11434 that speaks both the OpenAI and Anthropic API formats. Point any tool that expects a cloud API at localhost:11434/v1 and it generally just works.
Release cadence is roughly weekly; v0.34.0 landed on September 5th. There's now a paid cloud tier ($20/mo Pro, $100/mo Max) for running models you can't host, but everything local remains free.
What you give up: control. The K/V cache type is a server-wide setting rather than per-model, GPU layer placement is automatic, and it defaults to 4-bit quantization unless you specify a tag like :8b-instruct-q8_0. It's also explicitly not built for serving many concurrent users.
Mac users: check that your GPU is actually being used
Ollama on Apple Silicon now routes by file format — safetensors models go to the MLX engine, GGUF models go to llama.cpp's Metal backend. Both are accelerated, but a startling share of "Ollama is slow on my Mac" complaints turn out to be silent CPU fallback. Run ollama ps while a model is loaded and confirm it says GPU, not CPU, before you go benchmark-hunting.
LM Studio — start here instead if you want a GUI
LM Studio is the same job with the opposite philosophy: a real desktop application, currently at 0.4.24. The feature that earns its place is the model browser — it shows every available quantization of a model with a color-coded indicator telling you whether it will fit in your memory before you download 40GB. For anyone still building intuition about quantization, that single feature is worth the install.
It's been free for commercial use since mid-2025, no form to fill in. It also does speculative decoding with drafter models, connects to MCP servers, and exposes an OpenAI-compatible server on port 1234 — though unlike Ollama, that server is a toggle you have to remember to switch on.
Requirements worth knowing: Apple Silicon only on Mac (no Intel), AVX2 on Windows, and it's the better of the two on AMD hardware — its changelog is full of ROCm and Vulkan work.
llama.cpp — the thing underneath
Here's the part most guides skip: Ollama and LM Studio are both, in large part, llama.cpp wearing a coat. It's the engine that executes GGUF files in both, plus LocalAI and llamafile. Builds land daily.
You'd run it directly for one reason — control the wrappers hide. Per-process K/V cache settings, explicit layer offload, slot inspection, health endpoints, Prometheus metrics, continuous batching for real concurrency, and the ability to load any .gguf on disk without a registry blessing it. On AMD Strix Halo hardware specifically, running llama-server directly with the Vulkan backend is measurably the fastest path for generation-heavy work.
Skip it unless you have one of those needs. The wrappers exist for a reason.
vLLM — probably not for you, and that's fine
vLLM is the serious serving engine: PagedAttention, continuous batching, built for throughput across many simultaneous requests. It's the right answer for five to twenty concurrent users. For one person at a keyboard it is mostly downside.
The hard numbers make the case better than I can. On a DGX Spark running a 120B MoE model, vLLM's own published benchmark shows decode at roughly 23 tokens/sec — and model load times of ten to fifteen minutes, plus about 25 seconds of JIT warmup on a cold start. That's an excellent profile for a service that loads once and serves all day. It's a miserable one for someone who wants to ask a question.
Run vLLM when you're serving an agent fleet, a small team, or batch jobs. Otherwise the ten-minute load is the whole review.
The honorable mentions
MLX is Apple's own framework, and on Apple Silicon it beats llama.cpp on decode by roughly 1.4–1.8×, stretching to 3× on mixture-of-experts models. Its weakness is prefill — on one measured 650-token prompt, MLX spent 94% of its time there and came out slower overall than GGUF. If your work is long-context or RAG-shaped, that advantage narrows or reverses. Apple's M5 Neural Accelerators appear aimed squarely at this gap: reported time-to-first-token is about 4× faster than M4 while generation is only 1.19× faster.
Open WebUI is the frontend most people end up putting in front of Ollama — a ChatGPT-style interface for your local models. Note its license changed in 2025: you must keep its branding unless you have 50 or fewer users in a rolling 30-day window. A solo operator is comfortably clear of that.
Jan and GPT4All are both MIT-licensed desktop GUIs worth knowing if LM Studio's closed-source app bothers you; GPT4All has built-in document RAG.
Quantization: The Two-Minute Version
Quantization compresses model weights. Lower bits, smaller file, less memory, some quality loss. The entire decision comes down to one table — perplexity change against the full-precision baseline on a 7B-class model:
| Quant | Quality loss vs F16 | Verdict |
|---|---|---|
| Q8_0 | +0.0004 | Essentially lossless — and 29% slower than Q4_K_M |
| Q6_K | +0.0044 | Imperceptible |
| Q5_K_M | +0.0142 | Barely measurable — worth it for math and multi-step reasoning |
| Q4_K_M | +0.0535 | The default. Small but real; imperceptible for most chat and coding |
| Q3_K_M | +0.2437 | Noticeably degraded — only if nothing else fits |
Pick Q4_K_M and stop thinking about it. Move up to Q5_K_M if you're doing math or multi-step reasoning and have the headroom. Note the sting in the first row: bigger quants aren't free even when they fit, because generation speed drops as the file grows.
Two refinements worth knowing. Unsloth's "dynamic" quants (the UD- and _XL tags you'll see on Hugging Face) consistently deliver better quality per byte than the plain equivalents — they ran over 150 divergence benchmarks across 9TB of models to tune them. And the new FP4 formats (NVFP4, MXFP4) genuinely shrink files — a 27B model drops from about 17GB to 14GB — but whether they beat Q4_K_M on quality is, as of September 2026, still unsettled with no independent benchmark suite having answered it. Use FP4 to make something fit, not because you assume it's better. And note it only speeds things up on Blackwell hardware; on a 3090 you get the memory saving and nothing else.
Weights are not the whole memory bill
A 27B model at Q4_K_M is roughly 17GB of weights — plus about 1.5GB of runtime overhead, plus context. And context is not a rounding error: one 4B-class model goes from 6GB to 12.5GB when you open it up to a 128K window. Budget for the context you actually intend to use, not the weights alone.
What's Worth Running, September 2026
The single most important shift this year: mixture-of-experts models changed what unified memory is good for. A dense 70B model drags every weight through your memory bandwidth for every token — about 5 tokens/sec on a 128GB box, which is unusable. An MoE model of similar total size activates only a fraction of its parameters per token and streams at 70–100 tokens/sec on the same hardware. Same memory, twenty times the speed.
The rule follows directly: on unified-memory machines, buy MoE, not dense.
| Your memory | Run this | Roughly |
|---|---|---|
| 8GB | Qwen2.5-Coder 7B (autocomplete), Qwen3.5-9B (chat) | ~5–7GB at Q4 |
| 16GB | Qwen3.6-35B-A3B (3B active) | ~16–22GB at Q3/Q4 |
| 24–32GB | Qwen3.6-27B or Qwen3.8-27B; Gemma 4 26B-A4B | ~17–20GB at Q4 |
| 64GB+ | gpt-oss-120b (MXFP4), Qwen3-Next-80B-A3B | ~35–65GB |
| 128GB | Muse Glimmer 30B, Nemotron-3-Super-120B-A12B, DeepSeek V4 Flash at low quant | varies widely |
Three specific recommendations for most readers. Qwen3.8-27B (released August, Apache-2.0, 262K context) is the mainstream dense workhorse — if you install one model, make it this. Gemma 4 26B-A4B from Google gives you the inference speed of a 4B model with the capability of a 25B, which is exactly the MoE bargain described above. And Muse Glimmer 30B is Meta's return to genuine open weights — Apache 2.0, multimodal, built for agentic workflows, and it ships with a speculative-decoding drafter that reportedly triples throughput on a 5090.
One niche worth flagging for developers: if you want fill-in-the-middle autocomplete in your editor, you still need a Qwen2.5-Coder model. The Qwen3.x line dropped FIM support. The common pattern is running two models at once — a small one for autocomplete, a large one for agentic work.
The Install, By Platform
Just tell me what to do
Mac (any Apple Silicon): Install Ollama. Pull qwen3.8:27b if you have 32GB+, gemma-4-26b-a4b if you want speed. Add Open WebUI if you want a browser interface. Verify GPU usage with ollama ps. Consider MLX-format models if decode speed matters more than long prompts.
AMD Strix Halo: Ollama with the Vulkan backend for the easy path; llama-server with Vulkan directly for maximum speed. Linux only, and the kernel version matters — AMD requires 6.18.4+ (or the specific Ubuntu HWE builds). Keep BIOS dedicated VRAM small and raise the shared memory limit instead; this trips up nearly everyone.
NVIDIA (DGX Spark, RTX): Ollama or LM Studio for interactive use. vLLM only if you're serving concurrency and can live with the load times. FP4 quants are worth using on Blackwell cards and memory-savings-only on anything older.
Windows generally: LM Studio. The GUI advantage is real and the ROCm/Vulkan support is better than the alternatives.
The Mistake Almost Everyone Makes
It's buying hardware for dense models and then being disappointed. People read "128GB runs a 70B model" and imagine cloud-like speed; they get 5 tokens/sec and conclude local AI isn't ready. The memory holds the model. The bandwidth decides how fast it talks. Those are different constraints, and on unified-memory machines the second one is the binding one.
Load an MoE model of similar size on the same machine and it's twenty times faster. Nothing about the hardware changed. The bottleneck was always the model architecture.
So: install Ollama, pull Qwen3.8-27B at Q4_K_M, and see what your machine actually does before you spend another dollar on it. If it's fast enough, you're done — you have a private assistant with no per-token bill. If it isn't, now you know which number to shop for, which is more than most buyers ever figure out.
Mentioned in this article
Picking the box first? The local AI hardware table compares every option on memory, bandwidth and measured tokens per second.