Quantized Convolution, Calibration, and Quantization-Aware Training

Part 1 of this lesson derived the affine quantization map itself — how a scale and zero-point turn a continuous weight or activation tensor into an int8 grid, and why weights are usually symmetric and per-channel while activations are usually asymmetric and per-tensor. This half picks up exactly where that left off: once every operand of a convolution is an int8 integer, what actually happens inside the multiply-accumulate loop, how is the calibration range that the whole scheme depends on actually chosen, and when does any of this make the model faster at all.

1. Quantized Convolution: From Affine Algebra to Integer Arithmetic

1.1 Expanding the dot product

Recall the affine relationship from Part 1, applied to one input activation and one weight:

x = s_x(q_x − z_x) and w = s_w(q_w − z_w)

A single output element of a convolution (or a linear layer — the arithmetic is identical, only the indexing differs) is a dot product over the reduction dimension i — input channels times kernel height times kernel width for a conv, input features for a linear layer:

y = Σ_i x_i w_i

Substituting the affine relationships for x_i and w_i and pulling the two scales out of the sum, since s_x and s_w are constants that don't depend on i:

y = s_x s_w Σ_i (q_xi − z_x)(q_wi − z_w)

Expand the product inside the sum fully — this is just FOIL applied to (a − b)(c − d):

(q_xi − z_x)(q_wi − z_w) = q_xi·q_wi − q_xi·z_w − z_x·q_wi + z_x·z_w

Summed over all i, and letting N be the number of terms in the reduction (the kernel volume: input channels times kernel height times kernel width):

y = s_x s_w [ Σ_i q_xi·q_wi  −  z_w Σ_i q_xi  −  z_x Σ_i q_wi  +  N·z_x·z_w ]

Four separate sums. This is the fully general form, and it is exactly why real quantized-inference toolchains care so much about the symmetric-weights convention from Part 1's Section 3.1. Fix z_w = 0 — weights quantized symmetrically, which is standard practice — and two of the four terms vanish outright:

y = s_x s_w [ Σ_i q_xi·q_wi  −  z_x Σ_i q_wi ]

What's left is genuinely cheap. Σ_i q_xi·q_wi is the raw integer multiply-accumulate — the actual work the hardware's int8 MAC array does. Σ_i q_wi depends only on the weights, which are fixed after training and known at model-conversion time; it can be computed exactly once, offline, for every output channel, and folded directly into that channel's bias term. At inference time, the "extra" correction term z_x·(Σ_i q_wi) costs nothing beyond a single stored constant addition per output channel — no per-element work inside the accumulation loop at all. This is the concrete payoff of Part 1's symmetric-weights argument: it doesn't just save a little resolution, it deletes half the arithmetic from the integer kernel's inner loop.

1.2 Why accumulation must happen in int32

Here is the question the source outline poses and doesn't fully answer: individual operands are 8-bit, so why does the accumulator need to be 32-bit? Work the actual numbers.

After the zero-point subtraction, (q_xi − z_x) is no longer confined to the raw int8 range [−128, 127] — it spans the difference between two int8 values, which can be as wide as the full code range: q_max − q_min = 127 − (−128) = 255. The weight term q_wi, quantized symmetrically with the top code −128 conventionally left unused (Part 1, Section 3.1), stays within [−127, 127]. So the worst-case magnitude of a single accumulated term is:

max|term| = 255 × 127 = 32,385

Check what each accumulator width can hold:

int8 accumulator (range [−128, 127], max magnitude 127): overflows on the very first term. 32,385 is roughly 255 times larger than the accumulator's entire representable range. An int8 accumulator cannot even hold a single worst-case multiply-accumulate result, let alone a sum of them.
int16 accumulator (range [−32,768, 32,767]): a single worst-case term, 32,385, actually fits — barely, since 32,385 < 32,767. But it takes only one more term of the same sign and magnitude to break it: 32,385 × 2 = 64,770, which overflows the positive int16 ceiling by roughly a factor of two. Two multiply-accumulates, and a 16-bit accumulator is already wrong.
int32 accumulator (range roughly ±2,147,483,647): take a realistic mid-depth convolutional layer — a 3×3 kernel over 256 input channels, so N = 3 × 3 × 256 = 2,304 accumulated terms per output element. The worst-case sum, every term at maximum magnitude and identical sign, is 2,304 × 32,385 = 74,615,040. That consumes only about 3.5% of int32's positive range — enormous headroom for arithmetic that would have overflowed int8 immediately and int16 after two terms.

That headroom is exactly why "accumulate in int32" is the default in essentially every int8 inference kernel: it isn't a number pulled from convention, it's the smallest standard integer width with enough margin above the realistic worst case for typical reduction depths.

It's worth stress-testing that claim rather than accepting int32 as unconditionally safe, because it isn't. The worst-case bound scales linearly with N, so there is a reduction depth at which even int32 runs out of room. Solving N × 32,385 > 2,147,483,647 gives a crossover point around N ≈ 66,300 accumulated terms. That sounds enormous until you consider a wide early-stage or stem-style convolution — a 7×7 kernel over 2,048 input channels, N = 7 × 7 × 2,048 = 100,352 — whose theoretical worst-case sum is:

100,352 × 32,385 = 3,249,899,520

That number exceeds int32's positive maximum of 2,147,483,647 by roughly 51%. So the honest statement isn't "int32 is always enough" — it's "int32 comfortably covers the reduction depths that show up in the overwhelming majority of real conv and linear layers, with a margin that only gets tight for unusually wide reductions in the tens of thousands of taps." In practice this worst case essentially never triggers, because it requires every single accumulated term across tens of thousands of taps to land at maximum magnitude with identical sign — a statistically vanishing event for trained, calibrated weight and activation distributions that are roughly zero-centered rather than adversarially aligned. It's the reason some NPU and DSP MAC arrays provision accumulator registers wider than 32 bits for their largest supported reduction dimensions, rather than treating 32 bits as an unconditional guarantee.

1.3 Requantization: back to int8

The accumulator from Section 1.2 produces an integer, but that integer is not yet a valid int8 output — it's the raw material for a real value y, scaled by s_x s_w, that still needs to be quantized into the next layer's own int8 representation, using that next tensor's own calibrated scale s_y and zero-point z_y (chosen by calibration — Section 2). This step is requantization: converting an int32 accumulator, scaled by the product of the input scales, into an int8 code scaled by the output's own scale.

Starting from the corrected accumulator derived in Section 1.1 — call it A = Σ_i q_xi·q_wi − z_x·Σ_i q_wi, still an int32 integer — the real value it represents is y = s_x s_w × A. To express that same real value as an int8 code in the output tensor's own quantization scheme, apply the ordinary quantization formula from Part 1, Section 2.4:

q_y = clamp( round(y / s_y) + z_y,  q_min, q_max )

Substitute y = s_x s_w × A and collect the three scales into a single combined multiplier M:

M = (s_x · s_w) / s_y, giving q_y = clamp( round(M × A) + z_y, q_min, q_max )

Everything about calibration, weight quantization, and activation quantization ultimately funnels into this one multiplier M per output channel (per-channel weight scales, from Part 1 Section 4, simply mean M is computed once per output channel rather than once per tensor).

Worked example. Suppose this convolution's input activation x is the output of a preceding ReLU, so — following Part 1 Section 3.2's asymmetric convention for post-ReLU activations — its calibrated range starts at zero and its zero-point sits at the integer floor: s_x = 0.02, z_x = −128. The weight tensor is quantized symmetrically per Part 1 Section 3.1: s_w = 0.01, z_w = 0. This layer's output also feeds into another ReLU downstream, so its own calibrated output scale is likewise asymmetric: s_y = 0.5, z_y = −128. For one output element, suppose the integer kernel's raw MAC loop returns Σ_i q_xi·q_wi = 82,400, and this output channel's precomputed weight-code sum (Section 1.1) is Σ_i q_wi = 640.

Step 1 — apply the zero-point correction:

A = Σ_i q_xi·q_wi − z_x·Σ_i q_wi
  = 82,400 − (−128)(640)
  = 82,400 − (−81,920)
  = 82,400 + 81,920
  = 164,320

Step 2 — compute the combined multiplier:

M = (s_x · s_w) / s_y
  = (0.02 × 0.01) / 0.5
  = 0.0002 / 0.5
  = 0.0004

Step 3 — requantize:

q_y = round(M × A) + z_y
    = round(0.0004 × 164,320) + (−128)
    = round(65.728) + (−128)
    = 66 − 128
    = −62

−62 falls inside int8's [−128, 127] range, so no clamping is needed here — the output activation's calibrated scale was wide enough to accommodate this particular accumulator value without saturating. (Section 2.2 works through what happens when the calibrated scale isn't wide enough.) Cross-checking with the unfactored form confirms the algebra: y = s_x s_w × A = 0.0002 × 164,320 = 32.864, and round(32.864 / 0.5) + (−128) = round(65.728) − 128 = −62 — the same answer, reached either by scaling first and requantizing second, or by folding everything into M up front.

1.4 Fixed-point M in real kernels

The derivation above computed M as an ordinary floating-point number, but that's a pedagogical convenience, not what an integer-only accelerator actually does — multiplying by a float M would reintroduce a floating-point unit into a pipeline whose entire purpose was to avoid one. The standard trick, used throughout Google's gemmlowp integer-inference library and documented in Jacob et al.'s integer-arithmetic-only inference work (the same line of research Krishnamoorthi's whitepaper builds on), is to approximate M as a fixed-point integer times a power-of-two shift: M ≈ M0 × 2^(−n), where M0 is a normalized int32 value and n is a small integer shift count, both computed once offline from s_x, s_w, and s_y. At inference time, "multiply by M" becomes an integer multiply by M0 followed by an arithmetic right-shift by n — no float arithmetic anywhere in the entire int8-in, int8-out pipeline, which is precisely the property that lets integer-only accelerators skip a floating-point unit in silicon altogether.

2. Calibration: Choosing the Range Everything Else Depends On

2.1 What calibration actually decides

Weight quantization parameters can be computed by simply reading the trained weight tensor directly — the values are static and known in full the moment training finishes. Activations are a different problem: their values depend on whatever input the network happens to be fed, so there is no fixed tensor to inspect ahead of time. Calibration is the process of estimating an activation tensor's real-valued range — and therefore its scale and zero-point — by running a representative sample of real data through the float model and recording what values that tensor actually takes on. A typical calibrated range for a mid-network activation might look like [−2.4, 2.1]: not a theoretical bound, but an empirical observation over the calibration set.

The entire downstream scheme is only as good as this dataset. If the calibration data doesn't represent the true deployment distribution — wrong preprocessing, an unrepresentative sample of classes, a data split that misses an important operating condition — the resulting scale and zero-point are wrong for real deployment inputs, and every one of the failure modes below (saturation, coarse resolution, silent accuracy loss) follows directly. Calibration is therefore not a numerical footnote; it's an inference-engineering decision with the same weight as any other data-dependent design choice in the pipeline.

2.2 Min/max calibration and its outlier problem

The simplest calibration method observes the literal minimum and maximum activation values across the calibration set and plugs them directly into Part 1's scale/zero-point formulas. It requires no extra bookkeeping beyond running the data through the model and tracking two running numbers — and it is exactly as fragile as that simplicity implies, because a single rare extreme value can dominate the entire range.

Worked example. Suppose an activation tensor's typical values, across the vast majority of the calibration set, fall within [−2.4, 2.1] — the range given in the source outline. Now suppose exactly one sample in the calibration set (an unusual input, a transient numerical spike, a rare edge case) produces a single activation value of 41.7. Min/max calibration has no way to distinguish "this is a rare anomaly" from "this is the true range" — it simply takes the observed maximum.

Without the outlier, r_min = −2.4, r_max = 2.1:

s = (2.1 − (−2.4)) / 255 = 4.5 / 255 = 0.01764706

With the outlier, r_min = −2.4, r_max = 41.7:

s' = (41.7 − (−2.4)) / 255 = 44.1 / 255 = 0.17294118

That single outlier makes the scale 44.1 / 4.5 = 9.8× coarser — every bucket in the quantization grid is now almost ten times wider than it needs to be for the values that actually matter. Quantize a perfectly ordinary value, r = 1.5, under both scales and watch what that costs:

Scenario	`z`	`q = round(r/s) + z`	Dequantized	Error
Without outlier (`s ≈ 0.017647`)	8	93	1.500000	0.000000
With outlier (`s' ≈ 0.172941`)	−114	−105	1.556471	0.056471

(z and z' follow directly from Part 1 Section 2.4's formula, z = q_min − round(r_min/s), applied to each scale.) A value that quantizes perfectly under the clean calibration picks up an error of roughly 0.0565 — about 3.8% of the value itself — purely because one anomalous sample, contributing essentially zero probability mass to the true distribution, was allowed to set the scale for the entire tensor. Min/max calibration's failure mode is never really about the outlier's own error; it's about what the outlier does to everyone else's resolution.

2.3 Percentile clipping

The direct fix is to stop trusting the literal extremes. Percentile calibration sets r_max (and, symmetrically, r_min if the distribution is two-sided) to a high percentile of the observed |activation| distribution — commonly the 99.9th percentile — rather than the absolute maximum, deliberately discarding whatever lies beyond that cutoff.

Applied to the numbers above: if the 41.7 spike represents fewer than 0.1% of observed samples, a 99.9th-percentile calibration would set r_max ≈ 2.1 — the range representative of the bulk of the data — entirely ignoring the outlier's effect on the scale. That single rare value (and any future value that also exceeds 2.1) simply saturates: it clamps to code 127 and its dequantized reconstruction stops tracking its true magnitude. But every ordinary value keeps the fine s ≈ 0.017647 resolution computed in Section 2.2's clean case.

This is an explicit, deliberate trade: accept a large, bounded error on a small, rare slice of the distribution in exchange for a large resolution improvement on the common case. The percentile itself is a tunable hyperparameter — 99.9%, 99.99%, 99% — and the tuning question is exactly the tension it looks like: a tighter cutoff (lower percentile) buys more resolution for the bulk but pushes a larger fraction of samples into saturation; a looser cutoff protects more of the tail at the cost of coarser buckets everywhere.

2.4 Entropy / KL-divergence calibration

Percentile clipping picks a fixed cutoff percentage regardless of what the distribution actually looks like — 99.9% is 99.9% whether the tensor is sharply peaked or has a heavy, gradually decaying tail. Entropy calibration — the method TensorRT documents and defaults much of its INT8 calibration workflow toward — instead directly optimizes the quantity that actually matters: how much statistical information is lost by clipping and quantizing at a given threshold, measured by Kullback-Leibler divergence between the original float activation distribution and the quantized reconstruction.

The mechanism, following the approach NVIDIA's Szymon Migacz described in the original 2017 "8-bit Inference with TensorRT" work and that TensorRT's entropy calibrator still implements: collect a fine-grained histogram of activation magnitudes across the calibration set (TensorRT's implementation uses on the order of 2,048 bins spanning [0, max_observed]). For a sweep of candidate saturation thresholds — each one a bin boundary in that histogram — simulate what an int8 quantizer would actually do: merge every bin beyond the threshold into a single saturated bin (this is what clipping does to the tail), then re-bucket the kept range into the number of levels int8 actually provides (128, for the positive/negative code space). Reconstructing that coarse 128-level quantization back at the original histogram's fine resolution — by spreading each merged level's probability mass evenly across the fine bins it absorbed — produces a candidate "quantized" reference distribution. Computing the KL divergence between the original (clipped, renormalized) histogram and that reconstruction gives a single number per candidate threshold: how much information this particular choice of clipping range destroys. Sweep the candidates, take the threshold with the lowest KL divergence.

The intuition is worth making concrete with a small worked illustration — deliberately simplified (8 fine bins collapsing to either 4 or 2 coarse levels, rather than TensorRT's real 2,048-to-128 mapping), but computed with the actual KL-divergence formula, D_KL(P‖Q) = Σ_i P_i · log2(P_i / Q_i), so the numbers are genuine.

Take a toy 8-bin float histogram (bin values 0.5 through 4.0 in steps of 0.5, decaying probability mass):

P = [0.28, 0.22, 0.16, 0.12, 0.09, 0.06, 0.04, 0.03]

Coarser quantization — merge every 4 fine bins into 1 coarse level (2 levels total): level totals are 0.78 (bins 1–4) and 0.22 (bins 5–8); spreading each level's mass evenly back across its 4 constituent bins gives the reconstructed distribution Q₂ = [0.195, 0.195, 0.195, 0.195, 0.055, 0.055, 0.055, 0.055]. Computing D_KL(P‖Q₂) term by term and summing gives approximately 0.081 bits.

Finer quantization — merge every 2 fine bins into 1 coarse level (4 levels total): level totals are 0.50, 0.28, 0.15, 0.07; spread evenly across pairs gives Q₄ = [0.25, 0.25, 0.14, 0.14, 0.075, 0.075, 0.035, 0.035]. Computing D_KL(P‖Q₄) the same way gives approximately 0.015 bits.

Roughly a 5.5x reduction in KL divergence just from matching the reconstructed distribution more closely to the original's actual shape. This is a simplified analog of exactly the trade-off entropy calibration searches over — but where TensorRT holds the number of int8 levels fixed at 128 and instead varies the threshold (how much of the tail gets clipped into the saturated bin), the same underlying mechanism applies: a range that stretches to accommodate a long tail forces the same fixed code budget to cover more of the distribution's fine detail, which is exactly what raises the divergence in the toy example above. Entropy calibration doesn't need a human to hand-pick a percentile; it searches for the threshold that minimizes this divergence directly, which is why it tends to perform well specifically on activation distributions with heavy or unusual tails — the case percentile clipping's fixed cutoff handles only approximately.

2.5 Comparing the three methods

Method	How the range is chosen	Strength	Weakness
Min/max	Literal observed min and max	Simplest to implement; no histogram bookkeeping	A single outlier permanently sets the scale for the whole tensor (Section 2.2)
Percentile clipping	Fixed percentile (e.g. 99.9th) of `\|activation\|`	Cheap, robust to rare outliers, one tunable knob	The cutoff percentage ignores the distribution's actual shape — same percentile, different tensors, different amounts of information lost
Entropy / KL-divergence	Threshold minimizing KL divergence between float and quantized histograms	Adapts to each tensor's actual distribution shape; directly optimizes an information-theoretic loss metric rather than a proxy	More expensive (histogram collection and threshold sweep per tensor); more implementation complexity than a single percentile lookup

None of these dominates unconditionally — min/max is fine when a tensor's calibration data genuinely has no outliers (some weight-adjacent tensors behave this way), percentile clipping is a solid default for most activation tensors, and entropy calibration earns its extra complexity specifically on the tensors where distribution shape varies enough that a fixed percentile leaves accuracy on the table — which is a large part of why it's historically been TensorRT's recommended default for INT8 activation calibration in CNNs.

2.6 Static vs. dynamic quantization

One more distinction, drawn directly from how ONNX Runtime documents its own quantization workflow: everything in Sections 2.2 through 2.5 describes static quantization — calibration happens once, offline, using a representative dataset, and the resulting scale/zero-point pairs are baked into the model before deployment. At inference time there is zero additional cost to determine those parameters; they're just constants the runtime already has.

Dynamic quantization computes activation scale and zero-point on the fly, per inference, by observing the actual min/max of that specific run's activation values as they're produced — no calibration dataset required at all, and the parameters adapt perfectly to whatever the real input distribution turns out to be, batch by batch. The cost is a small but real per-inference overhead: computing a tensor's min/max (or a full calibration-style pass) every single forward pass, rather than once offline. ONNX Runtime's own guidance reflects the trade-off directly: static quantization is generally recommended for CNNs, where activation ranges are comparatively stable across inputs and the offline calibration cost amortizes over the model's entire deployment lifetime, while dynamic quantization is recommended for RNNs and transformer-style models, whose activation ranges (particularly around attention and sequence-length-dependent operations) can shift enough between inputs that a single static calibration pass captures them poorly.

3. Quantization-Aware Training

3.1 Why post-training quantization sometimes isn't enough

Everything in Section 2 is post-training quantization (PTQ): take a model that finished training with zero awareness that it would ever be quantized, and fit calibration parameters to it afterward. For ordinary int8 quantization with per-channel weights and reasonable calibration, this typically lands within one to two percent of float32 accuracy — the number Krishnamoorthi's whitepaper reports across a wide range of CNN architectures, and the same figure referenced in Part 1's Section 4.6.

But PTQ's accuracy gap tends to widen sharply once the quantization gets more aggressive: int4 or lower bit-widths, activation-heavy quantization on architectures with unusually wide dynamic range, or small mobile-scale models with little redundancy to absorb rounding noise in the first place. The underlying reason is structural, not incidental — the training process that produced these weights never once saw quantization noise during optimization. Gradient descent converged to whatever minimum was easiest to reach under exact float32 arithmetic, with no pressure whatsoever to avoid solutions that happen to rely on fine gradations quantization will later destroy. Quantization-aware training (QAT) answers this by not quantizing the model after training at all — it quantizes the training process itself.

3.2 Fake-quantization nodes in the forward pass

QAT inserts fake-quantization operations at every point in the computation graph where the deployed int8 model will actually quantize — after weight tensors, after activation tensors, wherever a real dequantize/quantize boundary would sit in the eventual integer graph. A fake-quant op computes exactly the round-trip quantize-then-dequantize operation from Part 1:

x_fq = s × ( clamp(round(x/s) + z,  q_min, q_max) − z )

The critical detail is what stays float and what doesn't: the underlying weight tensor being trained remains float32 throughout — the optimizer still updates full-precision weights every step. What changes is that the value flowing forward through the fake-quant op has been deliberately corrupted to look exactly like what it would look like after genuine int8 quantization and dequantization: rounded to a coarse grid, clamped at the calibrated range's edges. Every downstream computation — the actual convolution, the loss function — now sees quantization-perturbed values, so the training loss directly reflects how much the model would suffer under real int8 deployment. The scale and zero-point used inside these fake-quant nodes during training are typically derived from a running, moving-average observer of each tensor's min/max as training proceeds — the same conceptual calibration mechanism from Section 2, just continuously updated rather than computed in a single offline pass.

3.3 The straight-through estimator in the backward pass

There's an immediate mathematical problem with training through a fake-quant node: round() is a staircase function. Its derivative is exactly zero almost everywhere it's defined, and undefined at every integer boundary. Literal backpropagation through a true round() derivative would deliver a gradient of zero to every weight touched by a fake-quant node, on almost every step — no signal for gradient descent to act on, training effectively frozen at whatever the weights happened to be when fake-quant was inserted.

The fix used throughout the QAT literature and implemented in PyTorch's torch.ao.quantization fake-quantize modules is the straight-through estimator (STE): during the backward pass, treat the fake-quant op as if it were the identity function for any input value that fell inside the calibrated (unclamped) range, and pass zero gradient for any value that got clamped/saturated at the range's edges — since nudging an already-saturated value slightly further can't change the op's output at all. Formally:

d(fake_quant(x))/dx ≈ 1   if x is inside the calibrated range
                     ≈ 0   if x was clamped (saturated) at q_min or q_max

This is a deliberate approximation, not the true derivative — but it's a defensible one, precisely because a fake-quant op is close to the identity function almost everywhere it isn't clamping: it only adds bounded rounding noise, never a large discontinuous jump, to values inside the calibrated range. Using the identity function's gradient as a proxy for that bounded-noise operation gives the optimizer a slightly biased but genuinely useful signal, and empirically this specific approximation — not some more elaborate smoothed surrogate — is what makes QAT converge reliably in practice across essentially every framework that implements it.

3.4 Why QAT recovers accuracy PTQ loses

The mechanism connects directly back to Section 3.1's diagnosis. Because the loss function being minimized during QAT already includes quantization noise on every forward pass, gradient descent has continuous pressure to find weight configurations that are robust to that noise — further from quantization bucket boundaries where a small perturbation would flip the rounded code, less dependent on activation values that sit near the calibrated range's saturating edge, generally reshaping the solution around the constraint rather than discovering the constraint's cost only after the fact. PTQ, by contrast, hands a calibrator a model that was optimized under no such pressure at all; if that model's optimum happens to rely on distinctions finer than the quantization grid can represent, PTQ has no mechanism to fix it, because the weights are already fixed by the time calibration runs.

This gap between PTQ and QAT widens specifically as quantization gets more aggressive. At ordinary int8 precision, PTQ with good per-channel calibration is often good enough that QAT's extra cost isn't worth paying. At int4 and below, or for architectures and layers unusually sensitive to rounding noise, PTQ's accuracy loss can become severe enough to be deployment-blocking, while QAT — because it actively reshapes both the weights and the training trajectory around the quantization noise simultaneously, rather than discovering the noise's effect post hoc — routinely recovers most or all of that gap. The trade is real, though: QAT needs the full training pipeline, labeled data, and additional compute and wall-clock time that PTQ's single calibration pass doesn't. It's usually run as a short fine-tuning phase starting from an already-trained float checkpoint rather than training from scratch, but it is genuine retraining, not a one-shot conversion step.

4. Quantization Is Not Automatically Faster

4.1 The assumption that doesn't hold

It is tempting to reason: float32 → int8 is a 4-byte-to-1-byte shrink, therefore inference gets roughly 4x faster, or at least 2x. Neither conclusion is guaranteed. Whether quantization actually speeds anything up is entirely conditional on the target hardware and software stack, and this is precisely the roofline-model connection Part 1 established in Section 1.2: a memory-bound layer benefits from int8's bandwidth reduction regardless of compute throughput, while a compute-bound layer benefits only if the hardware has genuinely faster int8 arithmetic — and if neither condition holds, quantization can cost more than it saves.

4.2 No real int8 kernel means pure overhead

The first failure mode is the bluntest: if the inference runtime's kernel library has no actual int8 GEMM or convolution implementation for the target hardware, calling the "quantized" op doesn't skip any computation. A common fallback behavior is to upcast the int8 operands back to float32 and run the ordinary float32 kernel underneath. In that scenario, the model pays the cost of quantizing its inputs (round(r/s) + z, an extra pass over every element) and dequantizing them back to float, plus the exact same float32 multiply-accumulate work it would have paid running the original model directly — strictly more work, not less. This is exactly the caveat ONNX Runtime documents explicitly: quantization can fail to improve, and can actively regress, performance on hardware that lacks efficient low-precision instructions. Section 1.3's fixed-point M trick and Part 1's SIMD-packing and dedicated dot-product-instruction arguments (Section 1.3 there) only pay off if the target actually has that hardware path; assuming it does without checking is the single most common way quantization projects end up slower, not faster.

4.3 Dequant/requant sandwiching between unsupported ops

Real computation graphs are heterogeneous. Convolution, linear/matmul layers, and some elementwise ops typically have mature, well-optimized int8 kernels across most inference runtimes. Plenty of other ops — certain activation functions, layer normalization, softmax, custom or less common ops — often don't, or have int8 implementations with much weaker optimization than their float32 counterparts. When a graph alternates between int8-supported and int8-unsupported operators, the runtime is forced to dequantize the tensor to float32 immediately before the unsupported op, execute it in float, then requantize the result back to int8 before the next supported op can consume it.

Every one of those boundaries is a full extra pass reading and writing the entire tensor. This is the direct continuation of Lesson 2's graph-optimization argument: the entire point of operator fusion there was minimizing how many times a tensor round-trips through memory between kernels. Fine-grained mixed precision reintroduces exactly that cost, concentrated precisely at the boundaries where precision changes. If those boundaries are frequent — a model whose ops don't cleanly separate into "large int8-supported blocks" and instead interleave supported and unsupported operators throughout — the accumulated dequant/requant overhead can exceed whatever the int8 arithmetic saved on the supported ops, and the quantized model ends up measurably slower than the float32 baseline it was meant to accelerate.

4.4 Bandwidth savings require staying in int8 through the pipeline

Part 1's opening argument for quantization was substantially a bandwidth argument: a 4x smaller tensor is 4x fewer bytes crossing the memory bus. That argument is about the model at rest — the serialized weight tensors sitting in DRAM or flash before a single layer executes. Whether that saving actually shows up during execution is a separate question entirely, and it depends on whether the runtime keeps intermediate activations in int8 as they move between compute units, not merely whether the weights are stored compactly.

If Section 4.3's sandwiching is happening — activations getting dequantized to float32 and written back to memory before an unsupported op, then read back and requantized afterward — every one of those intermediate reads and writes moves four times more bytes than an int8-resident pipeline would have needed. The bandwidth savings promised at the storage level simply never appear in the runtime profile, because the bytes actually crossing the memory bus during inference were float32 the whole time except at the two endpoints. Quantization is a license to move fewer bytes and use cheaper integer arithmetic — not a guarantee that either happens. Whether the license gets exercised depends entirely on whether the specific combination of runtime, kernel library, and hardware target provides a complete, well-fused int8 execution path through the model in question, end to end, not just at the boundaries.

Quantization's speedup is conditional, not automatic, at every single layer of the stack: the hardware must have a faster int8 execution path, the kernel library must actually implement it rather than silently upcasting, the graph must keep enough consecutive operators in int8 to avoid dequant/requant sandwiching, and the runtime must keep intermediate activations resident in int8 rather than only the weights at rest. Skip any one of these and quantization becomes pure conversion overhead layered on top of the same float32 compute. The only way to know which regime a given model and target actually falls into is to profile it — never assume the byte-count arithmetic alone predicts the latency number.