Quantization Fundamentals: Affine, Symmetric, and Per-Channel

Lesson 2 was about graph-level optimization — operator fusion, constant folding, layout transforms — squeezing waste out of how a model executes without touching a single numerical value it computes. Now let's shrink the numbers themselves.

1. Why Shrink the Numbers at All

1.1 The four-byte tax

A trained neural network is, at the storage level, just a very large collection of floating-point numbers: weights, biases, and (transiently, during inference) activations. The default representation for those numbers on essentially every training framework is float32 — 4 bytes per value, following IEEE 754: 1 sign bit, 8 exponent bits, 23 mantissa bits.

That 4-byte width was never chosen for inference. It was chosen for training, where you need enough dynamic range and precision to accumulate millions of small gradient updates without the numbers collapsing to zero or exploding — a genuinely hard numerical problem. Inference is a completely different regime: the weights are already fixed, and you are evaluating a fixed function, not integrating a noisy optimization process. Carrying training-grade precision into that fixed-function evaluation is a tax you pay by default, not a requirement.

int8 needs 1 byte per value. Swap float32 for int8 and the raw storage for every weight tensor drops by:

BytesPerValue(float32) / BytesPerValue(int8) = 4 / 1 = 4×

A 4x reduction in model size sounds like a nice-to-have until you notice what it touches:

• Memory footprint — a 100 MB float32 model becomes a ~25 MB int8 model, which matters enormously on a microcontroller or phone with a memory budget measured in low tens of MB, not gigabytes.
• Memory bandwidth — every weight has to be streamed from DRAM (or flash, on an MCU) into the compute unit at least once per inference. A 4x smaller tensor is 4x fewer bytes crossing that bus per pass.
• Cache pressure — a 4x smaller working set is 4x more likely to actually fit in L2/L3, converting DRAM round-trips into cache hits (see Lesson 1, Part 7 of the companion Computer Architecture series for exactly why that gap is measured in hundreds of cycles).

None of this is exotic — it is the direct, mechanical consequence of storing the same count of numbers in a quarter of the bits.

A concrete, real-sized example. Take one mid-sized convolution layer — 128 output channels, 64 input channels, a 3×3 kernel, a shape that shows up constantly in ResNet-style backbones:

WeightCount = Cout × Cin × K × K
            = 128 × 64 × 3 × 3
            = 73,728 weights
 
Bytes(float32) = 73,728 × 4 bytes = 294,912 bytes ≈ 288 KB
Bytes(int8)    = 73,728 × 1 byte  =  73,728 bytes ≈  72 KB

Cross-reference that against the memory-hierarchy table from Lesson 1, Part 7 of the Computer Architecture series: a typical modern L2 cache is roughly 256 KB-1 MB. The float32 version of just this one layer's weights (288 KB) already spills past a 256 KB L2 before you account for anything else competing for that space — activations, other layers' weights, the rest of the process's working set. The int8 version (72 KB) fits comfortably inside the same L2 with room to spare. That is not an abstract "4x smaller" statement — it is the difference between this layer's weights being an L2-resident hit on every reuse and being an L2 miss that has to go all the way to L3 or DRAM, hundreds of cycles away, every single time they're touched.

1.2 The roofline connection: quantization is an arithmetic-intensity lever

Lesson 1 of this series introduced arithmetic intensity — operations performed per byte moved — as the number that predicts whether a kernel is memory-bound or compute-bound, and the roofline model as the picture that makes the tradeoff visible. Quantization is one of the most direct ways to move a kernel rightward on that plot, because it attacks the denominator of the ratio directly:

ArithmeticIntensity = Operations / BytesMoved

Switching a matmul's weight tensor from float32 to int8 does not change the operation count (still the same number of multiply-accumulates), but it cuts BytesMoved for those weights by 4x. For a weight-heavy, memory-bound layer — which describes most of the linear/conv layers in a typical CNN or transformer running on an edge device — that is a roughly 4x increase in arithmetic intensity from a single, mechanical change. On the roofline plot, that is a horizontal jump toward the ridge point, and for a kernel that started out memory-bound, a jump like that can translate almost directly into a proportional latency win, because the bottleneck was never the ALU in the first place — it was the wait for bytes to arrive.

Quantization is not primarily an accuracy trick that happens to also save memory. From an arithmetic-intensity standpoint it is a bandwidth optimization first: identical operation count, one-quarter the bytes, and therefore up to 4x the arithmetic intensity for memory-bound layers — before any change in compute throughput is even considered.

1.3 int8 also raises the compute roofline, not just the bandwidth slope

The bandwidth argument alone would be enough to justify quantization, but real hardware compounds it: many CPUs, GPUs, and NPUs execute int8 arithmetic faster than float32 arithmetic, independent of the bandwidth savings. This shows up in two concrete, real places:

• SIMD register packing. A 128-bit SIMD register holds four float32 lanes but sixteen int8 lanes — four times the elements per instruction, purely from the width arithmetic (128 bits / 32 bits = 4 vs. 128 bits / 8 bits = 16).
• Dedicated low-precision instructions. ARM's SDOT/UDOT (signed/unsigned dot-product, ARMv8.2 NEON) and Intel's AVX-512 VNNI vpdpbusd both compute packed int8 multiply-accumulates into int32 accumulators in a single instruction, something with no float32 equivalent of the same throughput. NPUs and edge accelerators typically go further, building their MAC arrays natively out of int8 (or even int4) multipliers, because an int8 multiplier is dramatically smaller in silicon area and power than an fp32 multiplier — full IEEE 754 float multiply requires exponent handling, mantissa alignment, and rounding logic that an integer multiplier simply doesn't need.

Both effects point the same direction: int8 execution moves the flat, compute-bound ceiling of the roofline plot upward, at the same time quantization moves memory-bound kernels rightward toward it. Krishnamoorthi's 2018 survey on quantizing convolutional networks reports this compounding effect concretely — 8-bit quantization delivering roughly 2-3x latency speedups on general-purpose CPUs and up to 10x on fixed-point SIMD DSPs purpose-built for int8 arithmetic (Qualcomm's Hexagon HVX being the example cited).

2. The Affine Quantization Formula, Derived From Scratch

2.1 The problem to solve

We have a real-valued tensor — say, a layer's weights — whose values span some continuous range [r_min, r_max]. We want to represent each value with a b-bit signed integer, whose representable range is [q_min, q_max] (for int8: q_min = -128, q_max = 127). We need a rule that is:

• Deterministic and invertible (approximately) — given a float, produce an integer; given that integer, recover something close to the original float.
• Linear — a straight-line mapping, so that integer arithmetic on the quantized values corresponds to (approximately) the same arithmetic on the real values. This is the property that lets a convolution or matmul be computed entirely in integer arithmetic and only converted back to real values once, at the end, rather than dequantizing every operand before every multiply.
• Parameterized by only two numbers per tensor (or per channel) — a scale and an offset — so calibration is cheap and the parameters are cheap to store and ship alongside the model.

That is exactly the shape of an affine function: real ≈ scale × integer + constant. This is affine quantization.

2.2 Setting up the affine map

Write the real value as r and its quantized integer representative as q. An affine (linear-plus-offset) relationship between them takes the form:

r ≈ s × (q − z)

where s (the scale) is a positive real number with units of "real value per integer step," and z (the zero-point) is an integer — specifically, the integer code that represents the real value 0.0.

Why must z be exactly representable in the integer grid, rather than merely a fitted real-valued offset? Because zero is not just another number in a neural network — it is the identity element for addition, and it shows up structurally everywhere: zero-padding at tensor borders, ReLU's floor, additive bias terms, and the implicit zeros in sparse activations. If the quantization scheme could not represent 0.0 exactly, every one of those structural zeros would pick up a small quantization error, and that error would then propagate and often compound through every layer that touches it. Forcing z to be an integer (not just a real-valued fitted constant) is precisely what guarantees r = 0 maps to q = z exactly, with zero rounding error, every single time.

2.3 Solving for the scale

To pin down s and z, anchor the affine line at the two ends of the range we're calibrating against: the smallest real value r_min should land at (or very near) the smallest integer code q_min, and the largest real value r_max should land at (or very near) the largest integer code q_max. That is a two-point line-fit problem, and the slope of a line through (r_min, q_min) and (r_max, q_max) is:

s = (r_max − r_min) / (q_max − q_min)

For int8, q_max − q_min = 127 − (−128) = 255, so the scale is simply the real-valued range divided by 255 — the number of distinct steps available. This single number, s, is the size of one quantization "bucket": every representable integer step corresponds to a jump of exactly s in real-value space. It is directly the resolution of the quantization scheme, and it is why the phrase "quantization error" almost always means "an error bounded by roughly s / 2" — the worst case for rounding to the nearest grid point is landing exactly halfway between two adjacent buckets.

2.4 Solving for the zero-point

Given s, invert the affine relationship to express q in terms of r:

r = s(q − z)
r / s = q − z
q = r / s + z

Since q must be an integer, round the right-hand side:

q = round(r / s) + z

This is the forward quantization formula. Reading the earlier line-fit equation, r = s(q − z), backwards gives the inverse, dequantization:

r ≈ s(q − z)

(The ≈ rather than = is the whole story of quantization error: round() is lossy, so recovering r from q never perfectly reconstructs the original value — it reconstructs the center of whichever bucket r was rounded into.)

Now solve for z itself, using the anchor condition that r_min should map to q_min:

q_min = round(r_min / s) + z
z = q_min − round(r_min / s)

Two important operational details, both of which come up constantly in real quantization pipelines and both of which foreshadow Part 2 of this lesson (calibration):

1. z must be clamped into [q_min, q_max] and stored as an integer of the same width as q. A zero-point of, say, 140 would be meaningless for an int8 scheme whose codes only go up to 127.
2. Any real value outside the calibrated [r_min, r_max] range must be clamped, not merely rounded — the full formula is q = clamp(round(r/s) + z, q_min, q_max). Every real-world quantization implementation includes this clamp; skipping it is a common source of silent overflow bugs when a later inference input happens to exceed the calibration range the scale/zero-point were fit against.

2.5 Fully worked example

Take a weight tensor whose calibrated real-value range is r_min = -2.5, r_max = 3.1, quantized to signed int8, q_min = -128, q_max = 127. Before running the numbers, here is the picture the formulas above are describing — a continuous real-value line laid on top of a fixed integer grid of 256 evenly spaced buckets:

Real line:   −2.5 ────────────────── 0.0 ─────────────── 1.75 ── 3.1
                │                     │                    │      │
                ▼                     ▼                    ▼      ▼
Int8 grid:    −128 ── ... ── −14(=z) ── ... ── 66 ── ... ── 127
             (q_min)        (zero-point)                  (q_max)
 
Each step on the int8 grid = one scale-width bucket, s ≈ 0.021961 real units wide.
r_min anchors to q_min.  r_max anchors to q_max.  r = 0.0 always anchors to z exactly.

The zero-point z is just wherever 0.0 happens to fall once the two endpoints are pinned down — in this case, 14 steps above q_min. Everything below now follows mechanically from the two formulas already derived.

Step 1 — scale:

s = (r_max − r_min) / (q_max − q_min)
  = (3.1 − (−2.5)) / (127 − (−128))
  = 5.6 / 255
  = 0.0219607843...

Step 2 — zero-point:

round(r_min / s) = round(−2.5 / 0.0219607843)
                  = round(−113.8392857)
                  = −114
 
z = q_min − round(r_min / s)
  = −128 − (−114)
  = −14

So this tensor's quantization parameters are s ≈ 0.021961, z = -14. Now quantize four specific values and dequantize them back, tracking the error at each step:

A few things worth pulling out of this table:

• Both endpoints hit their target codes exactly — -2.5 lands on q_min = -128 and 3.1 lands on q_max = 127, confirming the scale/zero-point derivation was solved correctly.
• Zero dequantizes to exactly zero. This is the property argued for in Section 2.2, and the table shows it isn't a coincidence — it's structural: whatever integer z turns out to be, s(z − z) = 0 exactly, every time.
• Every error is bounded by s / 2 ≈ 0.010980. Check: 0.003529 < 0.010980 and 0.006863 < 0.010980. This is the general guarantee of round-to-nearest quantization — no single value's error can exceed half a bucket width, and this bound is entirely a function of the scale, independent of which specific value you plug in.

The scale sets the resolution ceiling. Once s is fixed by the calibration range, no amount of clever rounding can push per-value error below roughly s/2 in the worst case. This single fact is the entire motivation for everything in Sections 3 and 4: every subsequent refinement (symmetric vs. asymmetric, per-tensor vs. per-channel) is really an argument about how to keep s as small as possible for the values that matter.

3. Symmetric vs. Asymmetric Quantization

The general affine scheme in Section 2 allows z to be any integer. Two important special cases show up so often in practice that they get their own names.

3.1 Symmetric quantization: forcing z = 0

Symmetric quantization fixes the zero-point at 0 and derives the scale from the single largest-magnitude value in the tensor rather than from separate min and max:

s = max(|r_min|, |r_max|) / q_max

Using q_max (rather than the full q_max − q_min span) as the denominator, and reserving the code -128 as unused, is the standard convention (used by both PyTorch's and TensorFlow Lite's weight quantizers) — it keeps the positive and negative sides perfectly mirrored: a value of +m and -m land on codes +c and -c for the same magnitude c, with no rounding asymmetry between the two signs. With z fixed at 0, the formulas from Section 2 collapse to:

q = round(r / s) and r ≈ s × q

No zero-point term anywhere — this is the entire appeal of symmetric quantization. In a quantized matmul, the zero-point of the weight operand would otherwise introduce an extra cross-term in the expanded dot product (the product (a - z_a)(w - z_w) has four terms once expanded, one of which is a z_a · z_w correction that has to be computed and subtracted back out). Pin z_w = 0 for weights and that whole correction term for the weight side vanishes, which is a real, measurable simplification in the integer kernel that actually executes the layer.

Worked example, reusing the same weight values as Section 2.5 (r_min = -2.5, r_max = 3.1) so the comparison to affine/asymmetric is direct:

max(|-2.5|, |3.1|) = 3.1
s_sym = 3.1 / 127 = 0.0244094488...
z = 0

Notice the cost of forcing symmetry on this particular data: 3.1 (the max) dequantizes perfectly, but -2.5 — which is not the largest-magnitude value, only the most negative one — now carries a larger error (0.010236, versus 0.003529 under the asymmetric scheme in Section 2.5). The scale itself is coarser too: s_sym ≈ 0.024409 versus the asymmetric s ≈ 0.021961, about 11% worse resolution. The reason is structural, not incidental: symmetric quantization must reserve code space for [-3.1, +3.1] even though this tensor's actual values never go below -2.5. Everything between code -128 and the code representing -3.1's mirror-of-2.5 is wasted range that no real value in this tensor will ever use.

3.2 Asymmetric quantization: z ≠ 0

Asymmetric quantization is simply the general affine scheme from Section 2 with no constraint on z — it uses the tensor's actual r_min and r_max, independently, which is exactly why it doesn't waste code space on a symmetric range the data doesn't occupy.

This matters most for activations, and specifically for anything downstream of a ReLU. A ReLU's output is, by definition, non-negative — max(0, x) cannot produce a negative number. So a typical post-ReLU activation tensor has a real range that looks like [0, r_max], entirely on one side of zero. Forcing symmetric quantization onto a range like that would be actively wasteful: the negative half of the int8 code space (-128 through -1) would sit completely unused, cutting the effective resolution roughly in half for no benefit, since no post-ReLU value will ever need a negative code.

Worked example. Take a post-ReLU activation tensor with calibrated range r_min = 0.0, r_max = 6.0 (a bound of this shape is common in mobile-oriented architectures that clip activations, e.g. ReLU6-style designs), quantized to signed int8, q_min = -128, q_max = 127.

s = (6.0 − 0.0) / (127 − (−128))
  = 6.0 / 255
  = 0.0235294118...
 
round(r_min / s) = round(0 / s) = 0
z = q_min − round(r_min / s) = −128 − 0 = −128

The zero-point here, z = -128, is exactly q_min — that's a direct consequence of the fact that this tensor's real minimum, 0.0, is required to land on the integer minimum. And notice the practical effect: the entire int8 code range, all 255 steps, is spent covering [0, 6.0], which is exactly the span the data actually occupies. Compare that to what symmetric quantization would have done with this same data: to keep the zero-point at 0, a symmetric scheme would need a range of [-6.0, 6.0] (since it must mirror the largest-magnitude value), doubling the covered span for the same 255 codes — i.e. roughly halving the resolution for values that, in reality, are never negative.

The wasted-code-space effect is easiest to see side by side, both schemes applied to this same [0, 6.0] post-ReLU data:

Half the symmetric grid — every code from −127 up to the code just below 0 — represents negative real values that a post-ReLU tensor will structurally never produce. Those codes aren't merely idle; their absence from the usable range is exactly why the symmetric scale (≈0.047244) comes out roughly double the asymmetric scale (≈0.023529) for the same underlying data. Coarser scale means a larger s/2 error bound (Section 2.5's guarantee) on every single value.

3.3 Why the choice differs between weights and activations

This is the crux of why real quantization toolchains (PyTorch's torch.ao.quantization, TensorFlow Lite's converter) don't pick one scheme and apply it everywhere — they systematically pick symmetric for weights and asymmetric for activations, for reasons grounded in the two tensors' actual statistical shape and in what the integer kernel needs to compute efficiently:

This asymmetry in choices is exactly the recommendation documented in PyTorch's own quantization guidance: symmetric-per-channel for weights (better accuracy than per-tensor for the reasons Section 4 works out numerically), and affine (asymmetric) per-tensor for activations, calibrated with a moving-average min/max observer since activation ranges shift slightly across different input batches.

Symmetric vs. asymmetric is not an arbitrary style choice — it is a bet about where the data actually lives. Weights are naturally near-symmetric and benefit from the hardware simplification of z = 0. Activations, especially anything past a ReLU, are naturally one-sided, and forcing symmetry onto one-sided data throws away resolution for no compensating benefit. Pick the scheme that matches the data's actual shape, not the scheme that's simplest to implement.

4. Per-Tensor vs. Per-Channel Quantization

4.1 The problem: one scale has to serve the worst-case channel

Everything in Sections 2 and 3 quantized an entire tensor with a single (s, z) pair — this is per-tensor (sometimes called per-layer) quantization. It is simple: one scale, one zero-point, stored once, applied uniformly.

The problem surfaces the moment different slices of a tensor have genuinely different dynamic ranges. In a convolutional layer, the weight tensor has shape [OutChannels, InChannels, KernelH, KernelW] — it is really a stack of OutChannels independent filters, each learned somewhat independently during training. It is entirely normal, especially after batch-norm folding merges each output channel's batch-norm scale directly into its convolution weights, for one filter's weight magnitudes to be an order of magnitude larger or smaller than another filter's, purely because that particular batch-norm channel had a very different learned scale. PyTorch's own quantization documentation calls this out explicitly as the reason per-tensor weight quantization "performs poorly" in practice — high variance in conv weights across channels, specifically from batch-norm folding.

A single per-tensor scale, derived from Section 2.3's formula, is forced to span the global min and max across every channel combined. That means the scale is set by whichever channel happens to have the largest range — and every other channel, especially the ones with much smaller natural magnitudes, gets quantized at a resolution far coarser than their own data would need, wasting most of their available code space on values they never produce.

Per-channel quantization is the direct fix: instead of one (s, z) pair for the whole tensor, compute an independent (s_c, z_c) pair for each output channel c, using only that channel's own min/max. Each filter gets a scale tuned to its own dynamic range.

4.2 Worked example: three conv filters with mismatched ranges

Take a (deliberately exaggerated, to make the effect unmistakable) 3-output-channel convolution weight tensor, with the true real-valued range of each channel's weights measured from calibration data:

Channel 0: range [−0.15, 0.20]   (max |w| = 0.20)
Channel 1: range [−1.80, 2.10]   (max |w| = 2.10)
Channel 2: range [−0.05, 0.08]   (max |w| = 0.08)

This is a realistic shape for a batch-norm-folded conv layer — one channel (channel 1) with a comparatively huge learned scale, and two channels an order of magnitude smaller. Use symmetric quantization (the standard choice for weights per Section 3) throughout, to int8 range q_max = 127.

4.3 Per-tensor quantization

The per-tensor scale must cover the global maximum magnitude across all three channels: max(0.20, 2.10, 0.08) = 2.10.

s_tensor = 2.10 / 127 = 0.0165354331...

Quantize one representative near-peak weight from each channel using this single shared scale:

Channel 2's error jumps out immediately: a 10.24% relative error on a weight that's already small, versus 0.08%-1.05% for the other two channels. The reason is visible directly in the resolution arithmetic: channel 2's entire true range is 0.08 − (−0.05) = 0.13 wide, but the per-tensor bucket size is s_tensor ≈ 0.016535. That means channel 2's whole dynamic range spans only:

0.13 / 0.016535 ≈ 7.9 quantization levels

Roughly eight distinct representable values for the entire span of channel 2's weights — because the scale is set by channel 1's much larger range, not channel 2's own.

4.4 Per-channel quantization

Now give each channel its own scale, derived from its own max magnitude:

s_0 = 0.20 / 127 = 0.0015748031...
s_1 = 2.10 / 127 = 0.0165354331...   (same as s_tensor — channel 1 is the dominant channel)
s_2 = 0.08 / 127 = 0.0006299213...

Quantize the same three representative weights, each with its own channel's scale:

4.5 Comparing the two schemes directly

Channel 1 is unaffected — it was already the channel setting the global scale, so giving it its own scale changes nothing. But channels 0 and 2, the ones whose natural magnitude is far below the tensor-wide maximum, both see dramatic error reductions once they're allowed their own resolution — channel 2's absolute error shrinks by roughly a factor of 39.

Per-tensor quantization error is dominated by the single channel with the largest dynamic range — every other channel inherits that channel's coarse resolution whether it needs it or not. Per-channel quantization decouples resolution from the outlier channel entirely, and the improvement scales directly with how mismatched the channels' natural ranges are. When channel magnitudes are already similar, per-channel buys almost nothing (see channel 1 above); when they're mismatched by an order of magnitude or more — the normal case after batch-norm folding — per-channel is often the difference between a quantized model that matches float accuracy and one that visibly doesn't.

4.6 Why per-channel is applied to weights, not (typically) to activations

Per-channel quantization is not free — it trades a single stored (s, z) pair for C of them (one per output channel), and more importantly, it changes what the integer kernel has to do: instead of one scalar rescale at the end of a convolution, the kernel applies a different rescale to each output channel's accumulator. For weights, this is cheap and natural: the per-output-channel scale is applied exactly once per output channel, after the (already channel-separated) accumulation for that channel completes — no extra cost inside the innermost accumulation loop itself.

Applying the equivalent granularity to activations is a much harder ask. An activation tensor's "channels" are typically the same as the convolution's output channels, but the values that would need separate scales are spread across every spatial position, and — critically — the inputs to the next layer's convolution are exactly this per-channel-quantized activation tensor. Because a convolution sums contributions across input channels, having a different scale per input channel inside a single dot-product term breaks the linear-integer-arithmetic property that made quantized matmul cheap in the first place (Section 3.1's motivation for eliminating cross-terms). This is precisely why per-channel quantization is, in essentially every mainstream framework, applied to weights (where PyTorch and TFLite both explicitly support it for conv and linear layers) while activations stay per-tensor: it captures nearly all of the accuracy benefit — Krishnamoorthi's whitepaper reports per-channel weight quantization combined with per-layer activation quantization landing within about 2% of float32 accuracy across a wide range of CNN architectures — without paying the much larger hardware cost of per-channel activation rescaling inside the accumulation loop.

Further Reading

• Krishnamoorthi, R., "Quantizing deep convolutional networks for efficient inference: A whitepaper" (2018) — the widely-cited survey behind the per-channel-weights / per-tensor-activations recommendation and the CPU/DSP speedup figures used in Section 1.3.
• PyTorch, Quantization documentation — the authoritative reference for affine vs. symmetric schemes and per-tensor vs. per-channel support in torch.ao.quantization.
• PyTorch blog, "Practical Quantization in PyTorch" — a more applied walkthrough of the same scale/zero-point mechanics with real observer configurations.
• Google AI Edge (TensorFlow Lite), "Post-training quantization" — documents the exact real_value = (int8_value − zero_point) × scale convention and the asymmetric-activations/symmetric-weights split used throughout this chapter.
• GeeksforGeeks, "Quantization in Deep Learning" — an accessible overview of per-tensor vs. per-channel tradeoffs.

Part 2 of this lesson picks up exactly where the integer math here becomes a running kernel: how a quantized convolution actually accumulates in int32 and rescales back to int8, how calibration ranges get chosen from real data in the first place, and what quantization-aware training changes about the picture entirely.