Model Compression: Pruning, Distillation, and Sparse Inference

Lesson 7 was about making a fixed model run faster on fixed hardware — kernel fusion, memory planning, graph rewrites, all of it downstream of a compiler that never touches a single weight value. This chapter changes what gets optimized: pruning and distillation make the model itself smaller, and sparse inference is the surprisingly hard problem of turning "smaller" into "actually faster."

1. Three Different Problems Wearing One Name

"Model compression" gets used as if it were a single technique, but it is really three separate problems that happen to get bundled together, and conflating them is the single most common source of confusion for anyone new to this material:

1. Pruning is a selection problem: given a trained model, which weights, channels, or structures can be removed with the least damage to accuracy?
2. Distillation is a training problem: how do you get a small model to learn a function that is almost as good as a large model's, using the large model itself as a richer source of supervision than the raw dataset provides?
3. Sparse inference is a systems and hardware problem: given that some weights are now zero, how — if at all — do you turn that into a real reduction in latency, memory traffic, or energy?

The reason these need separating is that the first two problems are solved almost entirely in the training loop, while the third is solved almost entirely in the kernel and the silicon. It is completely possible to do a beautiful job on problem 1 — a 90%-sparse model that is functionally nearly indistinguishable from the dense original — and get a zero percent speedup from it, because problem 3 was never addressed. That gap surprises almost everyone the first time they hit it, and it is the organizing fact of this entire chapter.

2. Pruning: Deciding What the Model Doesn't Need

2.1 The importance heuristic: weight magnitude

Pruning needs a rule for deciding which parameters matter. The cheapest and most widely used rule is magnitude: a weight with a value near zero contributes almost nothing to its layer's output no matter what activation flows through it, so removing it (setting it to exactly zero) should perturb the function the network computes by the least amount, on average, of any single-weight removal.

This is an approximation, not a theorem. It ignores how a weight interacts with the rest of the layer — a weight can be individually small but sit in a position where it cancels a large, otherwise-dominant term, and gating or normalization layers can make small weights load-bearing in ways magnitude alone doesn't see. More sophisticated importance scores exist: Optimal Brain Damage and Optimal Brain Surgeon use second-order (Hessian) information about how the loss curves around each weight; movement pruning tracks how far a weight moves during fine-tuning rather than its raw value; Taylor-expansion methods estimate the loss increase from removing a weight directly. All of these are more expensive to compute than "look at the absolute value," which is exactly why magnitude pruning remains the default first thing anyone reaches for, and the technique this section works through by hand.

2.2 Unstructured pruning, worked by hand

Take a small 4×4 weight matrix — think of it as a tiny fully-connected layer, 4 inputs mapping to 4 outputs:

W =
[  0.80  -0.05   0.30  -0.02 ]
[  0.01   0.60  -0.40   0.03 ]
[ -0.90   0.02   0.07   0.50 ]
[  0.04  -0.60   0.01   0.20 ]

Unstructured (also called fine-grained) pruning scores every individual weight by its absolute value, independent of which row or column it sits in, and zeros out whichever fraction of them falls below a threshold. To hit a target sparsity of 50% (8 of the 16 weights zeroed), sort all 16 magnitudes:

0.01, 0.01, 0.02, 0.02, 0.03, 0.04, 0.05, 0.07,   ← smallest 8 (zero these)
0.20, 0.30, 0.40, 0.50, 0.60, 0.60, 0.80, 0.90    ← largest 8 (keep these)

The threshold that separates the two halves is 0.07 — zero every weight whose absolute value is at or below it. Walking the matrix and applying that rule:

Pruned W (unstructured, 50% sparsity) =
[  0.80   0      0.30   0    ]
[  0      0.60  -0.40   0    ]
[ -0.90   0      0      0.50 ]
[  0     -0.60   0      0.20 ]

Look at where the zeros landed: every single row keeps a different pair of columns, and every column keeps a different subset of rows. There is no row you can delete entirely, no column you can delete entirely, no contiguous block you can carve out and skip. The sparsity pattern is a scatter of individually-placed holes with no shared structure between them — that word, unstructured, describes the layout of the zeros, not the aggressiveness of the pruning.

2.3 Structured pruning, worked on the same matrix

Structured pruning does not score individual weights at all — it scores whole structures (an entire output channel, an entire attention head, an entire filter) and removes the structure as a unit if its aggregate importance is low. Applied to the same 4×4 matrix, treating each row as one output channel's weight vector, score each row by its L2 norm:

||row 1|| = sqrt(0.80² + 0.05² + 0.30² + 0.02²) = sqrt(0.7329) ≈ 0.856
||row 2|| = sqrt(0.01² + 0.60² + 0.40² + 0.03²) = sqrt(0.5210) ≈ 0.722
||row 3|| = sqrt(0.90² + 0.02² + 0.07² + 0.50²) = sqrt(1.0653) ≈ 1.032
||row 4|| = sqrt(0.04² + 0.60² + 0.01² + 0.20²) = sqrt(0.4017) ≈ 0.634

To hit the same 50% sparsity target — but now measured in channels, not individual weights — drop the two lowest-norm rows (row 4 at 0.634 and row 2 at 0.722) and keep the two highest (row 3 at 1.032 and row 1 at 0.856):

Pruned W (structured, 50% channel sparsity) =
[  0.80  -0.05   0.30  -0.02 ]
[ -0.90   0.02   0.07   0.50 ]

This is not a 4×4 matrix with two blanked-out rows — it is genuinely a 2×4 matrix now. The output dimension of this layer shrank from 4 to 2. There is nothing left to "skip"; the removed structure simply does not exist in the computation graph anymore.

2.4 The single most important, most surprising fact in this chapter

Here is the fact that trips up nearly everyone encountering pruning for the first time: the unstructured, 50%-sparse matrix from Section 2.2 will typically run at close to zero speedup — sometimes literally the same latency — as the original dense matrix, on ordinary CPUs and GPUs, using ordinary dense GEMM kernels. Half the values are exactly zero, and the matmul takes the same time anyway. This is not a minor caveat; it is the fact that makes or breaks whether a pruning effort produces a faster model at all.

Recall the register-blocked GEMM microkernel from Lesson 4: the whole design of that kernel is to stream fixed-size, contiguous tiles of the operand matrices into SIMD registers in a predictable, unrolled access pattern, and to keep the FMA (fused multiply-add) pipeline saturated by issuing a steady stream of vector multiply-accumulate instructions with no branches and no data-dependent control flow. That kernel does not — and structurally cannot, without being rewritten — ask "is this element zero?" before deciding whether to load, multiply, and accumulate it. It touches every element of every tile in the block, every time, because that fixed, unconditional traversal order is exactly what makes the pipeline predictable enough to saturate in the first place. A zero sitting inside a dense tile still gets loaded from memory into a SIMD lane, still gets multiplied (a multiply-by-zero costs the pipelined FPU exactly as many cycles as any other multiply — there is no fast path for "times zero" in a hardware multiplier), and still gets added into the accumulator, contributing nothing to the sum but consuming exactly the same cycle as a nonzero would have.

To actually skip work at those zero locations you need two things simultaneously, and both are expensive:

• A different storage format. Dense row-major or column-major layout has no way to represent "this element doesn't exist" — it always occupies its slot. A format that only stores the nonzero values (compressed sparse row/column, or similar) has to additionally store where each surviving value belongs, typically as a row and column index per nonzero. At 50% sparsity, storing one index per retained float32 value can cost as much as the value itself, eating most of the memory savings the sparsity was supposed to buy.
• A different kernel. A kernel built to walk a sparse index list instead of a dense tile gives up the predictable, contiguous, register-blocking-friendly access pattern entirely. Memory accesses become data-dependent — where the next nonzero is depends on the sparsity pattern of this particular matrix, which the hardware prefetcher cannot predict the way it predicts a fixed stride. Branch prediction and SIMD lane utilization both degrade, because a SIMD instruction wants all its lanes doing the same, uniform operation, and an irregular nonzero pattern rarely fills all lanes cleanly.

Put those two costs together against the savings, and the breakeven point for unstructured sparsity turns out to be much higher than most people assume — general-purpose sparse BLAS kernels usually need to be well into the 90%+ sparsity range, on CPU or GPU, before they consistently beat a dense kernel doing 100% of the (mostly wasted) work. At the 50%-to-80% sparsity levels that typical magnitude pruning produces without destroying accuracy, unstructured sparsity is, on stock dense hardware, essentially a memory-footprint optimization (a sparse-encoded checkpoint is smaller on disk) with no guaranteed latency benefit at inference time — sometimes it is even slower, once index-decoding overhead is included.

Sparsity is a property of the data. Speedup is a property of the kernel and the hardware. A matrix can be 90% zeros and run at exactly the same wall-clock time as its fully dense counterpart, because a standard SIMD GEMM microkernel has no mechanism to notice — or benefit from — a value being zero. This is the fact every other claim in this chapter has to be checked against.

2.5 Why structured pruning doesn't have this problem

Structured pruning sidesteps the entire issue, because it never asks the kernel to skip anything. The pruned matrix from Section 2.3 is not a 4×4 dense matrix with some rows zeroed out that a clever kernel needs to detect and route around — it is unconditionally, structurally a 2×4 matrix. Feed it to the exact same dense GEMM kernel that ran the original 4×4 matrix, with no modification whatsoever, and it runs proportionally faster, because there is proportionally less arithmetic to do and proportionally less data to move. No sparse format, no index metadata, no specialized kernel, no hardware sparsity support of any kind is required — this is the property that makes structured pruning's speedup unconditional: it works on any CPU, any GPU, any NPU, any BLAS library, today, using code that predates the concept of pruning entirely.

The cost of that guarantee is granularity. Unstructured pruning can keep the single most important weight in an otherwise-unimportant row and discard the rest of that row's weights individually; structured pruning has to make an all-or-nothing decision about the entire row at once, even if 3 of its 4 weights are important and only 1 is dead weight. For a fixed sparsity target, structured pruning is therefore usually the more damaging choice to accuracy, weight-for-weight, precisely because it cannot be selective at fine granularity — and that tension between "coarse but genuinely fast" and "fine-grained but rarely fast on real hardware" is the whole reason Section 4 exists.

2.6 Comparing the two directly

2.7 The Lottery Ticket Hypothesis

A natural question, once you accept that magnitude pruning can remove most of a network's weights without destroying accuracy, is why that's even possible — why should a network trained with far more parameters than it "needs" end up with so much prunable slack? Frankle and Carbin's 2019 paper offers an answer they call the Lottery Ticket Hypothesis: a randomly-initialized, dense network contains a much smaller subnetwork — a specific pattern of surviving weights at their original initial values — that, if trained on its own from that same starting point, can match the full network's accuracy in a comparable number of training steps. They call this subnetwork a "winning ticket," and their algorithm for finding one, iterative magnitude pruning, is simple to state: train the full dense network, prune the smallest-magnitude weights, rewind the surviving weights back to their original initialization values (not their current trained values), and retrain from there — repeating the cycle over several rounds rather than pruning to the final sparsity target in one shot.

The implication for this chapter is that heavy pruning is not merely "damage control" — a large network apparently already contains most of what a much smaller network would need to learn on its own, and pruning's job is closer to finding that smaller network than to approximating the large one. That framing is also the conceptual bridge into Section 3: distillation is a second, quite different way of getting a small model's weights into a good configuration, one that doesn't start from the large model's own weight values at all.

3. Knowledge Distillation: Learning From a Teacher's Uncertainty

3.1 Why hard labels are a thin training signal

The standard way to train a classifier is against ground-truth ("hard") labels: an image of a cat is labeled cat, encoded for training as a one-hot vector — probability 1 on the cat class, probability 0 on every other class. That label is correct, but it is also nearly information-free about anything except the single right answer. It says nothing about how the image relates to the other classes: whether this particular cat photo happens to look a bit like a small dog breed, or nothing at all like a fox, or is a genuinely ambiguous, blurry edge case.

A large, well-trained teacher model, run on that same image, produces something far richer than a one-hot label: a full probability distribution over every class. A teacher might output something like 70% cat, 25% dog, 5% fox — and that 25%-on-dog is not noise, it is exactly the useful part. It tells you this particular image sits closer to the cat/dog decision boundary than a typical, unambiguous cat photo would, and that dog is a far more plausible confusion than fox is. Geoffrey Hinton called this extra information "dark knowledge" — knowledge about the shape of the decision boundary and the relative similarity structure between classes that a hard label, by construction, throws away entirely. Training a small student model to match that full distribution, rather than only the argmax, is the core mechanism of knowledge distillation.

3.2 Temperature scaling: making soft labels actually soft

There's a wrinkle: a confident, well-trained teacher's raw softmax output is often already extremely peaked — closer to 99.9% cat, 0.05% dog, and everything else near zero — because a well-trained network is, correctly, very sure of easy examples. A distribution that peaked doesn't actually carry much more information than a hard label did; the "dark knowledge" is technically present but numerically squashed down near zero, invisible to a loss function that has to work with floating-point probabilities.

The fix is temperature scaling of the softmax. Instead of computing the ordinary softmax over the teacher's logits z, divide every logit by a temperature T before exponentiating:

p_i = exp(z_i / T) / Σ_j exp(z_j / T)

At T = 1 this is just the ordinary softmax. As T grows past 1, dividing every logit by a larger number shrinks the differences between logits before they hit the exponential, which flattens the resulting distribution and pulls the smaller class probabilities up off the floor, revealing the relative ranking among them without changing which class is most likely. As T shrinks toward 0, the opposite happens — the distribution sharpens toward a hard one-hot at the argmax.

A concrete example makes the effect obvious. Take teacher logits for four classes, [cat = 4.0, dog = 2.0, fox = 0.5, other = -1.0]. At T = 1 (ordinary softmax):

exp(4.0) = 54.598   exp(2.0) = 7.389   exp(0.5) = 1.649   exp(-1.0) = 0.368
sum = 64.004
 
p(cat)   = 54.598 / 64.004 ≈ 0.853
p(dog)   =  7.389 / 64.004 ≈ 0.115
p(fox)   =  1.649 / 64.004 ≈ 0.026
p(other) =  0.368 / 64.004 ≈ 0.006

At T = 1, the distribution is already nearly hard: 85% on cat, and dog/fox/other are barely distinguishable from each other in absolute terms (0.115, 0.026, 0.006 — all small, all close to the floor). Now divide the same logits by T = 4 before applying softmax, giving scaled logits [1.0, 0.5, 0.125, -0.25]:

exp(1.0) = 2.718   exp(0.5) = 1.649   exp(0.125) = 1.133   exp(-0.25) = 0.779
sum = 6.279
 
p(cat)   = 2.718 / 6.279 ≈ 0.433
p(dog)   = 1.649 / 6.279 ≈ 0.263
p(fox)   = 1.133 / 6.279 ≈ 0.180
p(other) = 0.779 / 6.279 ≈ 0.124

The ranking is unchanged — cat is still most likely, other is still least — but now every class carries a numerically meaningful, learnable probability mass: 43% / 26% / 18% / 12%. This flattened distribution is what actually gets used as the training target for the student; the raw T = 1 softmax was too close to a hard label to teach the student anything a hard label wouldn't have. Typical distillation setups tune T somewhere in the range of 2 to 20, treating it as a hyperparameter alongside everything else.

3.3 The distillation loss

In practice, the student is trained against a weighted combination of two losses, not the soft targets alone — Hinton, Vinyals, and Dean's original formulation combines a standard hard-label cross-entropy term with a soft-target term:

L_total = α · CE(y_hard, student_softmax_T=1)
        + (1 − α) · T² · KL(teacher_softmax_T, student_softmax_T)

CE is ordinary cross-entropy against the true hard label, computed at the student's normal (T = 1) softmax, so the student is still anchored to getting the right answer. KL is a divergence between the teacher's and the student's temperature-softened distributions, both computed at the same elevated T, so the student is also pulled toward matching the teacher's full ranking over classes, not just its top pick.

The T² factor in front of the soft-target term is not decorative — it corrects for a real numerical effect. The gradient of the softened cross-entropy/KL term with respect to a logit scales roughly as 1/T², since dividing logits by T before the softmax also divides the resulting gradients by (approximately) T twice over. Without the T² correction, the soft-target loss would shrink toward irrelevance every time T is increased to make the labels more informative — exactly the opposite of what the temperature knob is supposed to control. Multiplying by T² keeps the relative magnitude of the hard-label and soft-label gradients roughly balanced as T is tuned, so α and T can be adjusted somewhat independently.

3.4 Why a distilled student can beat a same-size model trained from scratch

This is the counterintuitive payoff of the whole technique: a small model distilled from a strong teacher will often out-perform an identically-sized model trained directly on the same hard-labeled dataset, with no teacher involved at all. A few reasons compound to produce this result:

• Richer signal per example. A single training image, labeled only cat, gives one bit of "which class" information (relative to the other classes) in the hard-label setting. The same image, run through the teacher, yields a full calibrated ranking across every class — effectively many soft constraints extracted from one input, rather than one hard constraint. The student sees, implicitly, far more supervisory signal per gradient step than the raw dataset alone provides.
• Implicit regularization. Hard-label cross-entropy pushes a model's logits toward extreme confidence on the correct class and toward zero everywhere else — that pressure, applied directly to a small model with limited capacity, tends to produce overconfident, sharply-cut decision boundaries that overfit quirks of the training set. Soft targets are, by construction, less extreme, and training against them tends to produce a smoother, better-calibrated decision boundary, similar in spirit to label smoothing but derived from the teacher's actual learned uncertainty rather than an arbitrary smoothing constant.
• The teacher has already solved a harder optimization problem. A large model, trained directly on the data, can explore a far richer hypothesis space during training and settle into a better región of function space than a small model — with its far more limited capacity — could realistically find on its own, working from the same raw labels. Distillation lets the small model inherit the shape of that better solution — approximating the teacher's decision function — rather than having to rediscover it from scratch under its own capacity constraints.

This is not a hypothetical effect. DistilBERT, one of the most widely cited applications of the technique, distills BERT into a model with roughly 40% fewer parameters and about 60% lower inference latency, while retaining around 97% of BERT's language-understanding performance on the GLUE benchmark — a result that would be very difficult to reach by simply training a same-size Transformer from scratch on the same corpus with hard next-token or masked-token labels alone. The same pattern shows up constantly in production large-language-model pipelines today: a large "frontier" model is used to generate training targets (either full soft-label distributions, or simply high-quality generated responses in sequence-level or response distillation) for a much smaller, cheaper-to-serve model, precisely because the smaller model trained this way consistently outperforms the same architecture trained directly on raw human-labeled data alone.

3.5 What distillation does and doesn't fix

It's worth being precise about what distillation buys you relative to the other two techniques in this chapter. Distillation does not touch sparsity at all — the resulting student model is a smaller, fully dense network, and it gets its speedup the ordinary way any smaller dense model does: fewer parameters, fewer FLOPs, smaller GEMMs, all of which are unconditional wins on any hardware, for exactly the same reason structured pruning's wins are unconditional (Section 2.5). Distillation and pruning are frequently combined rather than treated as alternatives — for instance, distilling a model down in width and depth, and then structurally pruning the result further, or the reverse order, pruning first and using the pruned model's own unpruned self as a teacher ("self-distillation") to recover accuracy lost during pruning.

4. Sparse Inference Infrastructure: The N:M Middle Ground

4.1 The gap this section closes

Sections 2.4 and 2.5 established a genuine tension: unstructured pruning is fine-grained and accuracy-friendly but, on ordinary dense hardware, essentially useless for latency; structured pruning gets guaranteed real speedup but pays for it with coarse, all-or-nothing decisions that tend to cost more accuracy per unit of compression. For years, that tension looked like a fundamental tradeoff you simply had to pick a side of. N:M structured sparsity is the answer hardware vendors converged on to get most of both: sparsity fine-grained enough to preserve accuracy close to unstructured pruning's level, but constrained into a pattern regular enough that dedicated hardware can exploit it unconditionally, the way it exploits structured pruning.

4.2 What N:M sparsity actually constrains

The rule: within every contiguous group of M values along a chosen dimension of a weight tensor, exactly N are allowed to be nonzero, and the remaining M − N must be exactly zero. NVIDIA's Sparse Tensor Cores, introduced with the Ampere architecture (and carried forward through Hopper and Blackwell), implement the specific case of 2:4 sparsity: every contiguous group of 4 weight values must contain exactly 2 nonzero values.

Laid out visually, the contrast with the two previous patterns is stark:

Dense (no sparsity):
[ w  w  w  w | w  w  w  w | w  w  w  w | w  w  w  w ]
 
Unstructured sparsity (~50%, arbitrary positions):
[ w  0  w  0 | 0  w  w  0 | w  w  0  0 | 0  w  0  w ]
  no shared pattern between groups — position of zeros is unpredictable
 
Structured (channel) pruning (~50%, whole rows removed):
[ w  w  w  w | w  w  w  w ]     ← 2 of 4 original rows deleted entirely
(other 2 rows simply no longer exist in the tensor)
 
N:M = 2:4 structured sparsity:
[ w  0  w  0 | 0  w  0  w | w  w  0  0 | 0  w  w  0 ]
  ^^^^^^^^^^   ^^^^^^^^^^   ^^^^^^^^^^   ^^^^^^^^^^
  exactly 2      exactly 2    exactly 2    exactly 2
  nonzero of 4   nonzero of 4 nonzero of 4 nonzero of 4
  (position within each group of 4 is free to vary)

The key detail in that last diagram: 2:4 sparsity looks unstructured within any single group — the two surviving positions can be any 2 of the 4 slots, and different groups are free to have their nonzeros in different positions, exactly like Section 2.2's magnitude-pruned matrix. What makes it different from plain unstructured pruning is the fixed group size and fixed count: every group, everywhere in the tensor, has exactly 4 slots and exactly 2 survivors, with no exceptions. That regularity, applied uniformly and predictably at a small, fixed granularity, is precisely what a hardware designer can build dedicated silicon around.

4.3 Why 2:4 specifically is a pattern hardware can special-case

This is the question worth sitting with, because it is not obvious at first why one particular ratio would be "hardware-friendly" when arbitrary sparsity isn't. The answer comes down to counting how many distinct patterns the hardware actually has to be able to handle.

For a group of 4 values with exactly 2 required to be nonzero, the number of possible nonzero-position patterns is C(4,2) = 6 — there are only six ways to choose which 2 of 4 slots survive: positions {0,1}, {0,2}, {0,3}, {1,2}, {1,3}, or {2,3}. Six is small enough that a chip designer can build a compact, fixed selector — effectively a small multiplexer network — directly into the tensor core's datapath, one that reads a short metadata code and routes exactly the right two dense activation values to be multiplied against the two surviving weights, entirely in dedicated wiring, with no general-purpose gather/scatter engine and no data-dependent branching in the instruction stream. NVIDIA's Sparse Tensor Cores implement exactly this: the compressed weight tensor stores only the surviving nonzero values (roughly half the storage of the dense tensor) alongside a compact per-group metadata field recording which 2 of the 4 original positions they came from. A dedicated sparse matrix-multiply instruction — exposed at the PTX level as mma.sp and accessible through NVIDIA's cuSPARSELt library — consumes that metadata to select and pair up the correct dense-operand values on the fly, and executes the reduced 2-of-4 multiply-accumulate at the same fixed cadence the dense tensor core pipeline already runs at.

Now compare that to arbitrary 50% sparsity over the same group of 4 (or worse, over some much larger tile): if the nonzero count per group is not fixed, or the group size is much larger, the number of distinct patterns the hardware would need to route explodes combinatorially — a general-purpose crossbar capable of routing any subset of positions to any set of lanes is a dramatically more expensive, more power-hungry, and lower-clock-speed piece of silicon than a fixed 6-way selector wired for exactly one known pattern shape. This is the same underlying principle as Section 2.4's register-blocking argument, pushed down to the hardware level: a SIMD/tensor-core pipeline is fast precisely because its access and routing pattern is fixed and known ahead of time, and 2:4 sparsity is the largest, most flexible sparsity pattern that still keeps that routing pattern small and fixed enough to be worth building in silicon. It is, in a real sense, unstructured pruning's fine granularity and structured pruning's hardware-regularity, pushed as close together as the combinatorics of a compact selector will allow.

4.4 Real speedup versus the theoretical ceiling

Since exactly half of the multiply-accumulates in a 2:4-sparse matmul are structurally skipped, the theoretical throughput ceiling for the matmul itself is a clean 2x over the dense equivalent, on hardware that supports it. Achieved end-to-end model speedups in practice are consistently reported below that ceiling — commonly in the range of roughly 1.2x to 1.5x on real models — for reasons the roofline framing from Lesson 1/3 already predicts: not every layer in a real model is compute-bound on the tensor cores in the first place (memory-bound layers see much less benefit from halving MACs), non-matmul operations like normalization, activation functions, and elementwise ops are untouched by sparsity entirely, and the compression/metadata bookkeeping introduces its own small overhead. The 2x figure is a ceiling for the sparse GEMM operation specifically, not a promise about the wall-clock latency of an entire model.

4.5 Getting weights into a 2:4 pattern

Two general workflows are used to produce a 2:4-compliant model. The more common one — NVIDIA's Automatic SParsity (ASP) toolkit follows this path — trains a dense model normally, applies one-shot magnitude-based pruning constrained to the 2:4 pattern (within each group of 4, keep the 2 largest-magnitude weights, zero the other 2), and then fine-tunes the now-sparse model for a modest number of additional steps to recover any accuracy lost in the one-shot pruning step. The less common alternative trains the sparsity constraint in from the start or partway through training, letting the optimizer adapt to the constrained pattern gradually rather than having it imposed after the fact. Both paths converge on the same deployment artifact: a compressed weight tensor plus its metadata, ready for the sparse tensor core pipeline described above.

4.6 The full picture, side by side

5. Composing the Techniques

None of these three techniques compete with each other — they attack different axes of the same problem, and production compression pipelines routinely stack all of them together with quantization (Lesson 3) on top. A weight value's total "cost" comes from three roughly independent knobs: how many distinct values there are (pruning shrinks this), how good a smaller architecture's learned function can be made to be (distillation improves this), and how many bits each surviving value costs to store and compute with (quantization shrinks this). A typical modern edge or mobile deployment pipeline touches all three, usually in this order: distill down to a smaller architecture during training (since it needs gradient-based fine-tuning against a teacher), prune the distilled model — structurally if the deployment target is ordinary dense hardware, to a 2:4 pattern if the target has sparse tensor cores — then quantize the pruned model to int8 or lower as the very last step before compiling it (Lessons 6-7) for the target device. Order matters here mostly because pruning and distillation both want to happen while the model can still be fine-tuned with gradients, while quantization is cheapest and safest applied last, right before deployment, once the architecture and weight values are otherwise final.

Further Reading

• Hinton, G., Vinyals, O., and Dean, J., "Distilling the Knowledge in a Neural Network" (2015) — the foundational distillation paper; introduces temperature-scaled soft targets and the combined hard/soft loss used throughout Section 3.
• Frankle, J. and Carbin, M., "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (2019) — proposes that dense networks contain prunable "winning ticket" subnetworks and introduces iterative magnitude pruning with weight rewinding.
• NVIDIA Technical Blog, "Structured Sparsity in the NVIDIA Ampere Architecture and Applications in Search Engines" — the authoritative description of the 2:4 pattern and Sparse Tensor Core mechanics referenced in Section 4.
• NVIDIA Technical Blog, "Sparsity in INT8: Training Workflow and Best Practices for NVIDIA TensorRT Acceleration" — practical workflow for producing and deploying 2:4-sparse models, including the ASP one-shot-prune-then-fine-tune recipe described in Section 4.5.
• PyTorch Blog, "Accelerating Neural Network Training with Semi-Structured (2:4) Sparsity" — a more applied, framework-level walkthrough of the same 2:4 mechanics, including realistic end-to-end speedup figures.
• GeeksforGeeks, "Neural Network Pruning in Deep Learning" — an accessible overview of structured vs. unstructured pruning tradeoffs.

Lesson 9 turns to measurement itself: profiling and benchmarking an inference system properly enough to know whether any of the techniques in Lessons 2 through 8 actually helped, and how to avoid the classic mistakes — cold-start contamination, unrepresentative batch sizes, optimizing the wrong operator — that make a benchmark lie to you.