Open Weights or API? Choosing and Running Models

Every language model runs on somebody's hardware. Call a closed model through an API and the weights, the GPUs and the per-token price belong to the provider. Download open weights and the model becomes a file of numbers that you must fit into memory and stream through a chip once for every token you generate. The choice is partly about trust, control and capability, and partly plain arithmetic: bytes, bytes per second and money per month. The arithmetic starts with a rounding trick that stores each weight as a small integer plus a shared scale.

Closed, open weights and open source

A closed (proprietary) model is reached only through its provider's API and apps. The weights never leave the provider, and its terms decide what you may send and do. Examples include OpenAI's GPT models, Anthropic's Claude, Google's Gemini and Amazon's Nova. An open-weights model publishes its trained parameters for download, so you can run it wherever you like: examples include Meta's Llama, Alibaba's Qwen, DeepSeek, Mistral's open models, Google's Gemma, Microsoft's Phi and OpenAI's gpt-oss. Licences range from permissive (Apache-2.0, MIT) to custom licences with acceptable-use rules or conditions for very large companies. Read the licence before you build on a model.

Open weights are not the same as open source AI. The Open Source Initiative's Open Source AI Definition 1.0 (October 2024) asks for the freedoms to use, study, modify and share, and for the preferred form for making modifications: the parameters, the complete code used to train and run the system, and enough information about the training data for a skilled person to build a substantially equivalent system. Most popular open-weights releases stop at the weights and inference code; fully open projects such as Ai2's OLMo go much further. For deployment the sharper question is simpler: can you hold the weights yourself?

Which one, when

Neither side wins everywhere. Score your product on each row:

There are middle grounds. Hosted inference providers serve open-weights models per token, so you can start without hardware and self-host later with the same model. Several clouds offer closed models inside your own cloud tenancy, under your region and enterprise terms. Dedicated deployments (reserved capacity at a fixed price) exist on both sides.

Many systems do not choose at all. A router sends each request to the cheapest model likely to handle it; a cascade tries a cheap model first and escalates when a scorer judges the answer unreliable. FrugalGPT (Chen, Zaharia and Zou, 2023) learned cascades over commercial APIs that matched the best single model's performance at a fraction of its cost on the tasks studied. Whatever you pick, decide with your own eval suite on your own data: a leaderboard measures someone else's task. A sensible default path is to prototype on a strong API to learn what quality is possible, build the eval suite, then test open models against it, and move traffic when one passes and the privacy, control or cost case holds.

Running a model yourself

• llama.cpp runs models stored in its GGUF format on CPUs, GPUs and Apple silicon. When a model is too big for the GPU it can keep some layers in system memory, at system-memory speed.
• Ollama and LM Studio wrap local engines in a model library, an app or command line, and a local OpenAI-compatible server. MLX is Apple's array framework for unified memory; its mlx-lm package runs and fine-tunes models on a Mac.
• For many users on GPUs, vLLM (PagedAttention: the KV cache lives in fixed-size blocks, like virtual-memory pages, so little memory is wasted) and SGLang (RadixAttention: requests share cached prompt prefixes) batch requests continuously. Hugging Face's TGI is now in maintenance mode, and its documentation points new users to vLLM, SGLang, llama.cpp or MLX.

# One machine: a 4-bit GGUF file served on localhost:8080
llama-server -m model-Q4_K_M.gguf -c 8192 --port 8080

# A GPU server for many users, with an OpenAI-compatible API on port 8000
vllm serve <org>/<model> --max-model-len 8192 --gpu-memory-utilization 0.90

Both speak the same chat-completions protocol as most hosted APIs, so moving between a provider and your own server can be a configuration change. That portability is your best defence against lock-in.

As a rough guide from the GPTQ and AWQ papers and community testing, 8-bit is close to lossless, 4-bit costs a little quality (more for small models and for demanding work such as maths and code), and 3 bits or fewer degrade quickly. Treat that as a prior and measure on your eval.

Memory: weights, cache and quantisation

A model with $N$ parameters at $b$ bytes each needs $M_W=Nb$ bytes for weights. A 70B model takes 140 GB at 16-bit ($b=2$), 70 GB at 8-bit, and about 39 GB in a typical 4-bit format, which spends about 4.5 bits per weight once block scales are counted.

The Glimpse shows absmax quantisation. For a block $\mathbf x$, choose the step $\Delta=\|\mathbf x\|_\infty/127$, store $q_i=\lfloor x_i/\Delta\rceil$ as int8 with one scale per block, and reconstruct $\hat x_i=\Delta q_i$. Since $|x_i/\Delta|\le127$, no value overflows, and rounding moves each value by at most half a step: $|x_i-\hat x_i|\le\Delta/2$. For $\mathbf x=(0.3,-2.54,0.017)$, $\Delta=0.02$, $q=(15,-127,1)$ and the worst error is $0.003\le0.01$. The bound also shows why outliers hurt: one huge value inflates $\Delta$ for everything sharing its scale. Hence small blocks (tens of weights in GGUF's formats) and LLM.int8()'s trick of keeping rare outlier features in 16-bit.

Weights are not the only tenant. As Next-Token Prediction showed, decoding keeps a key and a value per layer and KV head for every token seen, so for $B$ sequences of $T$ tokens

$M_{KV}=2\,L\,h_{kv}\,d_h\,T\,B\,b_{kv}.$

An 8B model with 32 layers, 8 KV heads and $d_h=128$ stores $2\cdot32\cdot8\cdot128\cdot2=131{,}072$ bytes per token at 16-bit. At 64k tokens that is 8.6 GB, nearly twice the 4-bit weights. Add an allowance for activations and buffers (the lab uses 10%) and compare with the device.

Speed: decoding is memory-bound

To produce the next token of every sequence, a decoding step must read each active weight once, plus the KV cache. Memory delivers at most $\beta$ bytes per second, so a step takes at least (bytes read)$/\beta$ seconds, whatever the arithmetic units do. Each step yields one token per sequence, so

$\text{tokens/s per sequence}\le\frac{\beta}{N_{\text{act}}\,b+M_{KV}}.$

An 8B model at 4.5 bits (4.5 GB) on a 1,008 GB/s card is capped near 224 tokens/s for one user with a short context; on a 288 GB/s card, near 64. A 70B model at 4.5 bits on a 546 GB/s laptop: about 14. The arithmetic is not the bottleneck. A token costs about 2 FLOPs per active weight, while an 80 GB datacentre card performs roughly 990 dense 16-bit TFLOP/s against 3.35 TB/s of bandwidth, about 300 FLOPs per byte read.

Batching closes that gap: one read of the weights serves all $B$ sequences, so throughput grows almost linearly in $B$ until compute or the growing KV cache becomes the limit. That is why busy servers make tokens far more cheaply than a laptop serving one person, and why vLLM and SGLang fight so hard for KV-cache memory. Prefill, which processes the whole prompt at once, already reuses each weight across many tokens and is usually compute-bound.

A mixture of experts splits the two numbers. Memory follows the total parameters, because every expert must be resident; single-stream speed follows the active ones. A model with 47B total and 13B active parameters needs the memory of a 47B model and decodes like a 13B one. (At larger batches, different tokens choose different experts, so more weights are read per step.)

Cost: the break-even volume

At $V$ tokens a month, an API costs $pV$. Running locally costs $F+cV$: the fixed part $F$ is the hardware price divided by its useful life in months, plus hosting, maintenance and people's time; the marginal part $c$ is mostly electricity, power draw times price divided by tokens per hour. Local is cheaper exactly when $F+cV\lt pV$, that is, when $F\lt(p-c)V$. If $p\gt c$, divide by the positive $p-c$: owning wins exactly above the break-even volume

$V^*=\frac{F}{p-c}.$

If $c\ge p$, the right-hand side $(p-c)V$ is never positive, so there is no break-even at any volume.

A worked example, every number illustrative. A USD 8,000 machine over 36 months gives $F\approx222$ USD a month. Drawing 450 W at 0.25 USD per kWh while generating 200 tokens per second gives $c\approx0.16$ USD per million tokens. Against an API at 1.00 USD per million, $V^*=222.2/(0.844\times10^{-6})\approx263$ million tokens a month, about 100 tokens every second around the clock. The machine makes at most 518 million tokens in a 30-day month, so it breaks even at about 51% utilisation. At 0.30 USD per million, $V^*$ would be about 1.5 billion: more than this machine can produce at all.

Be honest about $F$ and $c$. One engineer-day a month of upkeep can easily triple $F$ here. Uptime may need a second machine. Idle power is real. If the local model fails more often, count the cost of those failures. And $V^*$ moves every time API prices fall. Low or uncertain volume favours the API; high, steady volume with a good-enough open model and the skills to run it favours owning.

Read beyond

Read the equation in context

Section 2.1 defines absmax quantisation with one constant for a whole tensor. Section 3.1 gives every row of the hidden states and every column of the weights its own constant (vector-wise quantisation). That is not enough at scale: beyond about 6.7B parameters, a few feature dimensions with huge magnitudes appear in every layer. Section 3.2 keeps those, about 0.1% of values, in 16-bit and multiplies the rest in 8-bit, which preserved 16-bit performance up to 175B parameters while roughly halving memory.

The quantisation and serving methods you will meet in every local stack come from four more papers:

GPTQ compensates each rounding error using second-order information; AWQ protects the roughly 1% of salient weight channels identified from activations. vLLM's PagedAttention cut KV-cache waste and raised throughput 2–4× over earlier systems at the same latency. Pope et al. model inference as compute time, memory time and communication between chips: every decoding step loads the parameters and KV cache from memory, small batches leave compute idle, and int8 weights save memory time in exactly that regime. They report 29 ms per token for small-batch generation on PaLM 540B.

Your turn

Plan three deployments. Fit each model onto its device by choosing a precision, watching the weights, KV cache and overhead fill the memory bar, and read off how fast bandwidth lets it decode. Then price the result: slide the monthly volume until owning costs the same as paying per token.

Coding problems