What the numbers on a model actually mean

Before a model sees your text, a tokenizer chops it into pieces from a fixed vocabulary, usually 100,000 to 250,000 entries. Common words get a single token. Rare or long words get split into several. The space before a word is usually glued onto the front of it, so " cat" and "cat" are different tokens.

• Rule of thumb: 1 token ≈ ¾ of an English word. 1,000 tokens ≈ 750 words ≈ 1.5 pages.
• Code, numbers, and most non-English languages use more tokens for the same amount of meaning.
• This is why models struggle to count the r's in "strawberry". They see a couple of chunks, not ten letters.

Under the hood, a language model is a long stack of matrix multiplications. The entries in those matrices are the parameters. Training nudges billions of them until the model gets good at predicting the next token. After training they're frozen, and running the model (called inference) means pushing your tokens through them.

Every parameter is a number that has to sit in memory, so the memory math is simple:

More parameters usually means a more capable model, but it isn't the whole story. Training data and technique matter a lot, and a recent 9B model can beat a 70B model from two years earlier on many tasks. Big dense models have also become rare: most large new releases are MoE.

Keep one more fact in mind for later: to generate each token, the model has to read (nearly) every parameter from memory once. That single fact explains most of the speed section.

Models are trained in 16-bit precision (usually BF16), which can represent tens of thousands of distinct values per weight. Most weights don't need that much detail. Quantization rounds each weight to one of a small set of levels: 8-bit allows 256 levels, 4-bit allows 16, 2-bit allows only 4.

Good schemes split the weights into small blocks (often 32 to 256 weights) and give each block its own scale, so the levels fit the local range of values. Those scales take space too, which is why a "4-bit" file usually works out to 4.5–5 bits per weight.

Names you'll see

• GGUF files (llama.cpp, Ollama, LM Studio): Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, IQ2_XS… The number is roughly the bits per weight.
• Other formats: AWQ, GPTQ, EXL2/EXL3, MLX (Apple), and FP8 / FP4 on newer GPUs.

How much quality do you lose?

• 8-bit: practically indistinguishable from the original.
• 6- and 5-bit: very close. Hard to notice in normal use.
• 4-bit (Q4_K_M): the popular sweet spot. Small but measurable loss.
• 3-bit and below: noticeably worse. Huge models tolerate it better than small ones.

A useful rule: at the same memory budget, a bigger model at 4-bit usually beats a smaller model at 8-bit.

In an MoE model, part of each layer (the feed-forward block, which holds most of the parameters) is split into many separate experts, anywhere from 8 to several hundred. A small router looks at each token and picks the few experts that should handle it. The rest sit idle for that token. Attention and a few other shared parts always run.

So an MoE model has two sizes that matter:

• Total parameters decide how much memory you need. All experts have to be loaded, because any token might need any expert.
• Active parameters decide how much work each token takes, and so how fast it runs.

Names often spell this out: Qwen3.6-35B-A3B means 35B total, 3B active. Under the hood it has 256 experts and uses 8 of them per token, plus one "shared" expert that always runs.

Totals can get fuzzy. Qwen3.8-Flash-Next has a 125B core, plus a 51B n-gram lookup table and a 4B MTP module, so you'll see it quoted as anything from 125B to 180B. DeepSeek V4.1 Flash similarly carries about 196B parameters of n-gram tables (its Engram module).

Common misconception

The experts aren't "the math expert" and "the poetry expert". Routing happens per token, per layer, and a single sentence bounces between many experts. Whatever specialization they develop is statistical and hard for humans to read.

On quality, an MoE model usually lands somewhere between a dense model of its active size and a dense model of its total size.

For every token, the model computes three vectors:

• Query: what this token is looking for.
• Key: what this token offers, used for matching.
• Value: the information this token passes along if it's picked.

A token compares its query against the keys of every earlier token, turns those scores into percentages, and takes a weighted blend of their values. A model runs dozens of these attention heads in parallel per layer, each tracking different kinds of relationships, across dozens of layers.

In chat models, attention is causal: a token can only look backward, never at tokens that come after it.

The cost: each new token compares itself against every previous token. The longer the context, the more work each step takes. The keys and values from all those earlier tokens are what the next section is about.

Without a cache, generating token number 1,001 would mean reprocessing all 1,000 tokens before it, then doing it again for token 1,002. With the cache, each new token computes its own key and value once, adds them to the pile, and reads the pile for attention.

The cache holds one key and one value per token, per layer, per KV head:

• Context window is the maximum number of tokens the model can handle at once: your prompt, the chat history, and its reply. Setting a 128k context in your app reserves room for a cache that big.
• GQA (grouped-query attention) lets several query heads share one KV head. Llama 2 7B (2023) had 32 KV heads; Llama 3.1 8B (2024) has 8, which makes its cache 4× smaller. They're old, but they show the jump clearly in the demo below.
• Hybrid attention only gives some layers full attention. Qwen3.8 27B has 64 layers, but 48 of them use a "linear attention" variant (Gated DeltaNet) that keeps a small fixed-size state instead of a growing cache. Only 16 layers keep a KV cache, so its cache is a fraction of what a 64-layer model would normally need.
• KV cache quantization stores the cache in 8-bit (or 4-bit), halving it or better, at a small quality cost.
• Other tricks: sliding-window attention, and DeepSeek's compressed attention. DeepSeek V4.1 Flash stores its cache in 4-bit and squeezes it to about 890 bytes per token, so even a 1M-token context needs under 1 GB.

This is also why models get slower as the conversation grows: every new token has to read the whole cache, on top of the weights.

Each layer takes a list of numbers per token (its input), multiplies it through the layer's weights, and hands a new list of numbers to the next layer. Those lists are the activations. For a model with a hidden size of 5,120, each token carries 5,120 numbers from layer to layer, and the feed-forward block briefly expands that to a much wider list in the middle.

Three kinds of memory are easy to mix up:

Why you'll run into them

• Prefill uses more. During prefill, hundreds of tokens go through each layer together, so the scratch space is much bigger than during decode. In llama.cpp this shows up as the "compute buffer", and the -ub (micro-batch) setting controls how many tokens go through at once: bigger is faster prefill but more memory.
• Training keeps them all. To learn, the model has to work backward through every layer, which needs every layer's activations. That's why fine-tuning a model takes several times the memory of running it.
• Quantization names. Labels like W4A16 or W8A8 mean "weights in 4-bit, activations in 16-bit" and so on. Activations are harder to squeeze than weights because they have occasional huge outlier values.

Discrete GPU + VRAM

VRAM sits right next to the GPU chip and is extremely fast: 1,000–1,800 GB/s on a high-end card. But there isn't much of it: 8–32 GB on consumer cards, 96 GB on workstation cards. Your regular system RAM is slower (about 80–100 GB/s for a typical dual-channel DDR5 desktop) and sits on the other side of the PCIe bus.

When a model doesn't fit, runtimes like llama.cpp split it: some layers on the GPU (-ngl), the rest on the CPU. Speed drops toward the speed of the slow part.

There's a good trick for MoE models: keep attention and shared layers on the GPU, and put only the expert weights in system RAM (llama.cpp's --n-cpu-moe or --override-tensor). Each token only touches a few experts, so not much has to be read from slow memory. This is how people run 100B+ MoE models on one gaming GPU plus a lot of RAM. Models with n-gram tables go a step further: the table can sit on an NVMe SSD.

Unified memory

Apple Silicon Macs, AMD Ryzen AI Max+ ("Strix Halo") machines and NVIDIA's DGX Spark have one shared pool that both CPU and GPU use, with 128–512 GB available and 250–820 GB/s of bandwidth. They can load models no consumer GPU can hold. The tradeoffs: generation is slower than the same model in VRAM, and prefill is noticeably slower because these chips have less raw compute.

On a Mac, the GPU can only use part of the memory by default (roughly two-thirds to three-quarters). You can raise the limit with the iogpu.wired_limit_mb setting.

Rule of thumb

• Small and mid-size dense models: a GPU with enough VRAM is fastest.
• Big MoE models: unified memory, or a GPU plus lots of system RAM with expert offloading.

Prefill (prompt processing)

All the prompt tokens are already known, so the model can process them in parallel with big, efficient matrix multiplications. The limit is raw compute. Benchmarks call this pp (prompt processing) speed, often hundreds to thousands of tokens per second. It sets your time to first token (TTFT): paste a 50-page document and this is the wait before anything appears. CPUs and Macs have less compute than big GPUs, so long prompts hurt most there.

Decode (generation)

Each new token depends on the one before it, so they have to come out one at a time. Every step reads all active weights from memory to do a fairly small amount of math. The limit is memory bandwidth, not compute. Benchmarks call this tg (text generation) speed.

That's why a tool like llama-bench reports two numbers, such as pp512 and tg128, and why a machine can be great at one and mediocre at the other.

During decode, the hardware streams every active weight through the processor once per token. So:

Example: an RTX 4090 moves about 1,000 GB/s. Qwen3.8 27B at Q4_K_M is about 16 GB. The ceiling is about 60 tokens/sec, and real software reaches maybe 60–80% of that.

This one formula explains a lot:

• Quantization speeds things up, because fewer bytes have to be read per token.
• MoE models are fast for their size, because only the active experts get read.
• Long chats slow down, because the KV cache gets read every token too.
• Not fitting in VRAM is brutal. Layers that spill into system RAM run at system-RAM bandwidth (often 10× slower), and the slow part dominates.

For a sense of scale: people read at roughly 5–10 tokens/sec. Around 20+ feels quick for chat. Coding agents and reasoning models, which generate lots of text you never read, benefit from much more. The next section lets you see the difference.

Roughly what each speed is like in practice:

• ~5 tok/s: a big model mostly running from system RAM. About reading speed, so chat is usable but long answers drag.
• ~20 tok/s: a mid-size model on a Mac or a squeezed GPU. Comfortable for chat.
• ~60 tok/s: a dense model that fits in a fast GPU. Faster than you can read.
• ~150 tok/s: a small-active MoE on a good GPU. Code appears in chunks.
• 1,000+ tok/s: specialized inference chips or diffusion models. The answer is basically just there.

Remember that decode is limited by reading weights, not by math. If one conversation needs the GPU to read 16 GB per token, ten conversations can share that same read: the GPU does ten tokens' worth of math on the weights it already pulled in. This is batching.

• Per-user speed drops slowly as you add users, because each one adds a bit of KV cache reading and math.
• Total throughput (all users' tokens per second added up) climbs steeply, until the GPU's compute becomes the limit.
• Memory caps it. Every conversation needs its own KV cache. Long contexts mean fewer users fit.

In practice

• llama.cpp (llama-server): --parallel N (-np) sets how many conversations ("slots") run at once. Depending on version and flags, the context size you set is either split between slots or shared by them, so raise it when you add slots.
• vLLM, SGLang: built for this. Continuous batching slots new requests in as soon as others finish, and paged KV caches avoid wasting memory.
• Batching helps MoE models less: different conversations pick different experts, so more weights get read per step.
• Running agents or batch jobs? Several parallel requests can finish far sooner than the same requests one after another.

Recall the speed formula: decode speed ≈ memory bandwidth ÷ active weights. A GPU keeps its weights in HBM, stacks of memory chips sitting next to the processor. Even a GB300 reads about 8 TB/s.

SRAM is memory built into the processor itself, the kind GPUs use for their small caches. It's dramatically faster, but it takes up a lot of chip area, so there's very little of it. The SRAM approach is to spread the model across enough chips that every weight lives in SRAM, and the bandwidth problem mostly goes away.

Groq: lots of small chips

Groq's LPU has about 230 MB of SRAM per chip and no HBM at all. A big model is split across hundreds of chips wired together, each holding a slice. Its compiler plans every data movement ahead of time, with no caches or guesswork, so timing is predictable and latency is very low.

Cerebras: one giant chip

Cerebras uses an entire silicon wafer as a single chip. The WSE-3 has about 900,000 cores and 44 GB of SRAM on one wafer, with on-chip bandwidth measured in petabytes per second. Bigger models are split across several wafers. In independent benchmarks it's usually the fastest for a single user: around 1,700+ tok/s on gpt-oss-120b.

Others

• SambaNova mixes SRAM, HBM and regular memory in tiers, so one system can hold more models.
• Google TPUs, Amazon Trainium and similar chips use HBM like GPUs. They compete on cost and scale more than on single-user speed.
• Transformer-specific chips (such as Etched's) hard-wire the transformer's math for efficiency, at the cost of flexibility.

The tradeoffs

• Capacity is expensive. Hundreds of chips per model means these providers offer a shorter list of models, and long contexts need room for the KV cache too.
• Speed per user vs cost per token. GPUs serving many users at once (batching) are often cheaper per token. SRAM chips win when one user's speed matters: coding agents, reasoning models, voice assistants that must answer instantly.
• Precision. To fit, models are sometimes served at reduced precision. Check what a provider runs if small quality differences matter to you.

Speculative decoding

A small, fast draft model guesses the next few tokens. The big model then checks all the guesses in a single pass, the same way it handles a prompt during prefill. It keeps every guess up to the first wrong one, and replaces the wrong one with its own token. Wrong guesses waste a little compute but never change the output: you get exactly what the big model would have produced alone.

Multi-token prediction (MTP)

Instead of a separate draft model, the model is trained with extra prediction heads that guess the tokens 2, 3 or more steps ahead. DeepSeek-V3 popularized this, and many newer open models ship with MTP layers. Qwen3.8-Flash-Next, for example, includes a 4B MTP module. At inference time those heads can act as a built-in draft model. Support varies: some runtimes use the MTP layers, others skip them and run the model normally.

How much faster?

It depends on how often the guesses are right. Predictable text like code, JSON, or rewriting a document you pasted can see 2–3× speedups. Creative writing sees much less. There are also draft-free tricks that reuse text already in the prompt, covered in n-grams.

A 2-gram (bigram) is two tokens in a row, like "peanut butter". A 3-gram (trigram) is three, like "peanut butter and".

The old way: counting

Early language models just counted. After "peanut butter and", the word "jelly" showed up most often in the training text, so that's the prediction. It works for common phrases and falls apart quickly: most longer runs of words never appear in any training text, so the counts have nothing to say. Neural networks replaced this because they generalize to sentences they've never seen.

Back again: n-gram speculation

When you ask a model to edit code or summarize a document, its answer often repeats chunks of your prompt. N-gram (or "prompt lookup") speculation takes the last few tokens the model wrote, finds the same sequence earlier in the context, and offers whatever came next as a draft. No draft model is needed, so the guesses are free. llama.cpp and vLLM both support this kind of drafting.

Back again: n-gram embeddings

Some new models add a big lookup table keyed on short runs of recent tokens, which gives the model extra stored knowledge for common phrases. The idea is to let a cheap table handle memorizing, so the network itself can spend its effort on reasoning.

• DeepSeek V4.1 Flash uses DeepSeek's Engram module: about 196B parameters of tables, split across two layers. For each token, it hashes the last 2, 3 and 4 tokens, looks up a few rows, and blends them into the model.
• Qwen3.8-Flash-Next has a 51B-parameter n-gram table on top of its 125B core.

Looking up a few rows is almost free compared to running the network, so these tables add a lot of memory but little work per token.

Put the table on your SSD

Here's the useful part for running these models at home. Which rows get fetched depends only on which tokens came before, not on anything the model computes, so the runtime knows what it needs ahead of time. It can fetch the rows early, while the GPU is still busy with the previous layers. That makes it practical to keep the table somewhere much slower and cheaper than VRAM:

• NVMe SSD: the popular choice. SemiAnalysis served DeepSeek V4.1 Flash with its ~189 GiB of Engram tables memory-mapped from a local NVMe drive. That's ~189 GiB that doesn't need to fit in VRAM or RAM at all.
• System RAM: faster, if you have room. SGLang can keep Engram tables in host memory to free GPU memory for the KV cache.
• Tiered: the frequently used rows cached in VRAM or RAM, and the long tail of rare phrases on the SSD.

Compare that with ordinary weights: putting a dense layer or an MoE expert on an SSD is painfully slow, because the model has to read it every time it's needed and can't know in advance which experts it will pick. N-gram tables are the rare case where SSD-speed storage barely costs you speed.

Autoregressive (almost every chat model)

Predict a token, add it to the text, predict the next one. Everything earlier in this guide (decode, KV cache, speculative decoding) is about making this loop faster. Its weakness is that it's strictly one after another: a 1,000-token answer needs 1,000 steps, and the model can't go back and fix an early word.

Diffusion

The model starts with a fixed-length block of "masked" placeholders. Each step, it predicts every position at once and locks in the ones it's most confident about. After a handful of steps, the whole block is filled. Because many tokens appear per step, diffusion models can be very fast, and they can revise earlier parts of the text, which helps with things like filling in the middle of code.

Examples: Inception Labs' Mercury, Google's experimental Gemini Diffusion, and open research models such as LLaDA and Dream. Some newer designs mix the two ("block diffusion"): they write left to right in blocks, and fill each block with diffusion.

The tradeoffs

• Quality has trailed the best autoregressive models, though the gap has been shrinking.
• Output length is usually set in advance per block, and tooling like llama.cpp support is far less mature.
• Image generators like Stable Diffusion and Flux use the same idea on pixels. That's where the name comes from.

Images

A vision encoder cuts the image into small square patches (often 14 or 16 pixels across) and turns each patch into a vector. Neighbouring patches are usually merged (2×2 is common) to save tokens. A small projector then maps those vectors into the same space as the model's text tokens, and the language model reads them like any other tokens. Higher resolution means more patches, which means more tokens and more prefill time.

In llama.cpp, the vision part ships as a separate mmproj file that you load next to the main GGUF.

Video

Video is handled as a series of images, sampled at a low frame rate (often 1–2 frames per second), usually at a lower resolution, and often with neighbouring frames merged. Even so, a few minutes of video can run to tens of thousands of tokens.

Audio

Sound is turned into a spectrogram (a picture of which frequencies are present over time), and an audio encoder, often based on Whisper, turns slices of it into tokens: typically tens of tokens per second of audio. Models that can also speak back (often called "omni" models) add a part that turns output tokens back into sound.

What this means for you

• Images and video eat context and make prefill slow, especially on Macs and CPUs.
• Downscaling an image before sending it can save a lot of time, if fine detail doesn't matter.
• Small text in screenshots needs high resolution to read, which costs more tokens.

The discovery

Inside a model, each token is represented by a long list of numbers (its activations), which you can think of as a point in a space with thousands of dimensions. In 2024, researchers showed that for many chat models, whether the model is about to refuse comes down to one particular direction in that space. Push the activations along it and the model refuses harmless requests. Remove it and the model stops refusing. The name is a mix of "ablation" (removing a part to see what changes) and "obliterate".

How it works, in three steps

1. Collect examples. Run the model on two sets of prompts: ones it answers normally and ones it refuses, and record the activations at some layer.
2. Find the direction. Average each set. The difference between the two averages points from "answering" toward "refusing". That's the refusal direction.
3. Edit the weights. Adjust every weight matrix that writes into the model's main data stream so its output has nothing along that direction (this is called orthogonalizing). From then on, no layer can push the model toward refusing.

No training data or retraining is needed. It's a one-time edit that takes minutes, and the result is an ordinary model file you can quantize and run like any other.

What it doesn't do, and the side effects

• It doesn't add knowledge or skills. The model only stops declining. It's no smarter, and it still gets things wrong.
• Quality often drops a little. The direction isn't perfectly clean, so removing it can also nudge unrelated behavior: slightly lower benchmark scores, more made-up answers, or weaker reasoning. Some releases run a short fine-tune afterwards to recover ("healing"), and newer variants try to edit more gently.
• It's not all or nothing. Refusal isn't purely one direction, so some refusals or lectures can survive, and some models lose useful pushback too, like correcting a false premise.
• The guardrails are gone. The model's built-in safety behavior was part of what got removed, so responsibility for how it's used moves to whoever runs it. Model licenses and acceptable-use terms still apply.

Related: steering

The same trick works for other behaviors. Instead of editing the weights permanently, steering adds or subtracts a direction while the model runs, to make it more formal, more concise or more upbeat, for example. Abliteration is the permanent version applied to one behavior.