NPU Architecture, Dataflow, and Hardware/Software Co-Design

Part 2 spent its energy on the MCU end of the deployment spectrum, where the entire discipline was making a general-purpose core survive a job it was never built for. Part 3 closes Lesson 10 at the opposite end of that spectrum: MCUs run tiny models on general-purpose cores — NPUs run models on hardware built for exactly this one job, and this part is about what that hardware actually looks like, and the compiler decisions it forces on everything upstream of it.

1. The Systolic Array: The NPU's Actual Compute Primitive

Start with what an NPU is not. It is not a CPU core with a wider ALU, and it is not a GPU streaming multiprocessor with different scheduling. Both of those are still, fundamentally, instruction-issue machines: some control unit fetches an instruction, decodes it, and dispatches it to an execution unit, and that cycle repeats — with heavy amortization through superscalar issue, SIMD width, and warp-level parallelism, but the shape of the machine is still "fetch work, then do work." A systolic array throws that shape away for the one operation that dominates neural network inference — the multiply-accumulate — and replaces per-instruction issue with a wave of data physically flowing through a fixed grid of arithmetic units.

The name is deliberate. H.T. Kung's 1982 paper, which coined the term, drew the analogy directly from cardiac physiology: in a systolic array, data flows from memory in a rhythmic fashion, passing through many processing elements before it returns, the way blood is pumped through the body and returns to the heart. Kung's actual argument was narrower and more useful than the metaphor suggests — he was arguing for a specific design discipline for special-purpose hardware, built on four requirements: the array should be simple and regular (cheap to design, easy to scale), it should achieve high concurrency (many operations in flight simultaneously, since that's the only way to buy real throughput once single-unit clock speed stops improving), its inter-cell communication should be simple (nearest-neighbor only, no global wiring that gets more expensive as the array grows), and — the requirement that matters most for this lesson — computation should be balanced against I/O, because a design that has to fetch a new operand from memory for every arithmetic operation it performs is a design that will always be memory-bandwidth-bound, no matter how many ALUs it has.

That last requirement is the one a matmul satisfies almost perfectly, and it's why systolic arrays and neural network accelerators found each other. A matmul reuses every input operand many times — each weight in a GEMM's B matrix participates in every output column, each activation in A participates in every output row — so if the hardware can be built to hold an operand in place and reuse it across many multiply-accumulates before it ever has to be re-fetched, the memory traffic per unit of arithmetic collapses. A systolic array is the mechanism that makes that reuse a structural property of the hardware layout, not something a program has to engineer through careful addressing.

1.1 The mechanics, precisely

A systolic array for matrix multiplication is a two-dimensional grid of processing elements, or PEs, wired only to their immediate neighbors — north, south, east, west. Each PE holds three pieces of local state at any given cycle: a stationary operand (in the weight-stationary design this lesson focuses on, a single weight value, loaded once and held for the duration of a tile), an operand streaming in from one neighbor, and a partial sum streaming in from another neighbor. On every cycle, a PE that has valid inputs does exactly one thing: it computes one multiply-accumulate using its stationary weight and the operand currently at its input, adds that product to the partial sum arriving from its neighbor, and then forwards both the streaming operand and the new partial sum onward to the next PE in line, one hop, arriving one cycle later.

No instruction is fetched for that multiply-accumulate. No address is computed for it. The PE always does the same operation, every cycle it has valid data — the entire "program" is the fixed wiring of the grid plus the schedule of when data enters from the edges. This is the fundamental trade the systolic design makes: it gives up the flexibility of a general instruction stream in exchange for eliminating essentially all of the fetch/decode/dispatch overhead that a general-purpose core pays per operation, and it gives up random-access memory traffic in exchange for the guarantee that every operand, once loaded, gets reused for as many cycles as it stays resident.

The subtlety that makes the grid actually compute the right answer — rather than a scrambled sum of misaligned terms — is the skew. Activations don't enter the array all at once; they enter from the west edge one row at a time, but each row's stream is delayed relative to the row above it, by exactly the number of cycles it takes a partial sum to travel one hop south. That delay is what synchronizes a given activation's horizontal journey through the array with the vertical journey of the partial sum it needs to be added to, so that by the time a term reaches the PE where it needs to be summed with the term above it, both values arrive in the same cycle. Section 1.2 works through exactly why that timing has to be one cycle per row, by tracing a real example end to end.

1.2 Worked trace: a 2×2 weight-stationary array

Take the smallest non-trivial case: a 2×2 grid of PEs computing Y = Wᵀ X for

W = | 1  2 |      X = | 5  6 |
    | 3  4 |          | 7  8 |

Row i of W and X corresponds to the reduction dimension (call it the input-channel axis); column j of W corresponds to the output axis. Weight w_ij is loaded once into PE(i,j) before any activation moves, and it never moves again for the duration of this tile:

Weights loaded once, before cycle 1 (weight-stationary — fixed for the tile):
 
  PE(1,1): w11 = 1        PE(1,2): w12 = 2
  PE(2,1): w21 = 3        PE(2,2): w22 = 4

X streams in from the west edge, one column of X per time-step, with row 2's stream delayed by exactly one cycle relative to row 1's — the skew from Section 1.1:

  cycle:         1     2     3     4
  row 1 west-in: 5     6     -     -
  row 2 west-in: -     7     8     -

Partial sums enter from the north edge as 0 — the top row has no PE above it to sum against. Every PE computes psum_out = psum_in + weight × activation_in on any cycle it has both inputs, then forwards the activation east and the partial sum south, each arriving at the neighbor one cycle later. Tracing all four PEs, cycle by cycle:

Cycle 1
  PE(1,1): west_in=5, north_in=0  →  psum = 0 + 1×5 = 5
           → forwards activation 5 east  (reaches PE(1,2) at cycle 2)
           → forwards psum 5 south       (reaches PE(2,1) at cycle 2)
  PE(1,2): idle — nothing has reached it yet
  PE(2,1): idle — nothing has reached it yet
  PE(2,2): idle
 
Cycle 2
  PE(1,1): west_in=6, north_in=0  →  psum = 0 + 1×6 = 6
           → forwards activation 6 east (→ PE(1,2), cycle 3)
           → forwards psum 6 south      (→ PE(2,1), cycle 3)
  PE(1,2): west_in=5 (from PE(1,1) @ cycle 1), north_in=0
           →  psum = 0 + 2×5 = 10
           → forwards psum 10 south (→ PE(2,2), cycle 3)
  PE(2,1): west_in=7 (row 2's own stream), north_in=5 (from PE(1,1) @ cycle 1)
           →  psum = 5 + 3×7 = 26
           → forwards psum 26 south — no row 3 below: this value IS an output
           >>> Y[j=1, t=1] = 26 <<<
  PE(2,2): idle
 
Cycle 3
  PE(1,1): idle — only 2 activations exist in this stream, both issued
  PE(1,2): west_in=6 (from PE(1,1) @ cycle 2), north_in=0
           →  psum = 0 + 2×6 = 12
           → forwards psum 12 south (→ PE(2,2), cycle 4)
  PE(2,1): west_in=8 (row 2's own stream), north_in=6 (from PE(1,1) @ cycle 2)
           →  psum = 6 + 3×8 = 30
           → forwards psum 30 south — output
           >>> Y[j=1, t=2] = 30 <<<
  PE(2,2): west_in=7 (from PE(2,1) @ cycle 2), north_in=10 (from PE(1,2) @ cycle 2)
           →  psum = 10 + 4×7 = 38
           → forwards psum 38 south — output
           >>> Y[j=2, t=1] = 38 <<<
 
Cycle 4
  PE(2,2): west_in=8 (from PE(2,1) @ cycle 3), north_in=12 (from PE(1,2) @ cycle 3)
           →  psum = 12 + 4×8 = 44
           → forwards psum 44 south — output
           >>> Y[j=2, t=2] = 44 <<<

Collecting the four outputs into a matrix, indexed Y[j, t], gives exactly Wᵀ X:

Y = | 26  30 |
    | 38  44 |

Checked by direct computation: Y[1,1] = w11·x11 + w21·x21 = 1·5 + 3·7 = 26, and the remaining three entries follow the same pattern. Every one of the eight scalar multiply-accumulates in this 2×2×2 matmul happened inside a PE that never issued an instruction, never computed an address, and never touched a general-purpose register — it happened because the right two numbers arrived at the right PE on the right cycle, and that timing was guaranteed entirely by the skew.

Two things about this trace generalize directly to real hardware. First, the array pays a one-time pipeline fill-and-drain cost — here, three of the array's four cycles are spent partially idle while the wave of data fills the grid and drains back out, and only cycle 2 has every relevant PE active. For a real array computing a 128×128 weight tile against a stream of thousands of activation vectors, that fill/drain cost is a few hundred cycles paid once, amortized over the thousands of cycles that follow where every PE is busy every cycle — which is exactly why systolic arrays are throughput machines that need a long enough stream to be worth their fixed latency, and why a single-inference, batch-1, short-sequence workload structurally under-utilizes the same hardware that crushes a long, high-batch one. Second, once the pipeline is full, this design retires one fully-reduced output column per cycle out of the bottom edge, for a K-deep reduction, using a grid of only K rows — the systolic array turned a K-term reduction that would otherwise take K sequential accumulate operations into K PEs' worth of spatial parallelism, retiring in one cycle what a scalar accumulator would take K cycles to do serially.

Real NPU matrix units follow this exact design, at far larger scale. Google's TPU, the paper that put weight-stationary systolic arrays into a real, characterized production chip, built its matrix multiply unit as a 256×256 grid of 8-bit multiply-accumulate units — 65,536 MACs wired exactly the way this trace shows, loaded with weights once and streamed with activations for as many cycles as the batch of work justifies (Jouppi et al., 2017).

2. On-Chip SRAM Scratchpads vs. Caches: Explicit Management as the Whole Point

Every systolic array needs a local memory to feed it — weights have to come from somewhere before they're loaded into the grid, and activations have to come from somewhere before they stream across the west edge. On an NPU, that local memory is SRAM sitting on the same die as the compute array, and it is worth being precise about why it is called a scratchpad and not a cache, because the distinction is not terminology — it's a difference in who is in control, and it is the single most consequential architectural decision shaping how an NPU compiler has to be written.

A cache is transparent. The hardware decides what's resident based purely on the addresses the running program happens to touch: a tag array, a replacement policy (LRU or an approximation of it), and an associativity structure work together so that, from the program's point of view, every load or store just goes to "memory" — whether that access hits in L1, misses down to L2, or goes all the way to DRAM is invisible to the instruction stream, only visible in latency. The program has no direct control over occupancy. It can influence the cache indirectly — Lesson 2's tiling and Lesson 4's register blocking are both, at bottom, exercises in shaping access patterns so the hardware's replacement policy behaves the way you want — but the compiler never actually writes "put this array into L1 and keep it there." It writes loads and stores to ordinary addresses and hopes the resulting access pattern is one the cache's automatic policy handles well.

An NPU scratchpad inverts that completely. It's addressed directly, as an explicit region the compiler owns byte for byte — there are no tags, no associativity, no replacement policy, and critically, no hit/miss distinction at all. Either the compiler's schedule has already placed the data there before the array needs it, or it hasn't, and if it hasn't, the systolic array simply stalls waiting for a DMA transfer to bring the tile in from DRAM. This is the same shift Lesson 5's memory-planning material made for activation buffers on a CPU/MCU runtime — deciding, ahead of time, exactly where every tensor lives — just enforced now by hardware that has no fallback path at all if the compiler gets it wrong.

That inversion is simultaneously a burden and an opportunity, and it's worth being honest about both sides. The burden: there is no safety net. A CPU cache degrades gracefully when a working set is slightly larger than L1 — some accesses spill to L2, latency goes up, throughput drops, but the program still runs correctly and reasonably fast. A scratchpad has no such graceful degradation. If the compiler's tile size doesn't fit the physical SRAM budget, the options are a compile-time rejection (the tile literally does not fit, full stop), or a fallback to streaming the tile from DRAM on every access — which doesn't just slow the array down, it can defeat the entire premise of building a systolic array in the first place, because the array's whole throughput advantage assumed operands were resident and reused, not re-fetched every cycle. Getting scratchpad tiling right is not a performance nice-to-have on this hardware; it's close to a correctness-adjacent requirement for the design to deliver anything like its advertised throughput.

The opportunity is the mirror image of that same fact. Because there's no tag array, no associativity metadata, and no cache-line granularity mismatch between what the tensor's access pattern actually wants and what the hardware happens to fetch on a miss, a correctly tiled scratchpad program achieves reuse that a general-purpose cache can only approximate through indirect, statistical means. There's no compulsory-miss overhead from a cache line that's wider than the useful data in it, and no capacity wasted on maintaining replacement-policy state. Every one of the scratchpad's bytes is either exactly the operand the array needs next, or it's something the compiler explicitly decided to keep resident for later — nothing is held "just in case," the way a cache holds recently-touched lines on the chance they'll be reused.

This is worth tying directly to material this reader has already seen twice in this series, because NPU scratchpad tiling is not a new idea — it's the same blocking hierarchy from Lesson 4, applied one more level up. Lesson 4's register-blocking derivation sized a MR × NR micro-kernel tile so it could stay resident in the CPU's register file for the entire KC-deep reduction, because registers are the fastest, smallest, and most explicitly-addressed level of that hierarchy. Lesson 2's graph-level tiling made the analogous decision one level further out, sizing tiles to fit an on-chip cache or SRAM budget so an intermediate tensor didn't have to round-trip through DRAM between producer and consumer. An NPU scratchpad sits at exactly the same conceptual layer as Lesson 2's tiling target, just with the cache's implicit management replaced by the compiler's explicit management — the tension is identical (keep a tile resident long enough to amortize its load cost, without asking for a tile larger than the resident storage can hold), only the mechanism enforcing that tension has changed from a hardware policy to a compiler decision. The TPU's own scratchpad is a concrete instance of the scale this operates at: a 24-MiB Unified Buffer alongside 28 MiB of total software-managed on-chip memory feeding that 65,536-MAC array (Jouppi et al., 2017) — every byte of which some compiler pass decided, explicitly, to place there.

3. Dataflow Taxonomies: Weight-, Output-, and Activation-Stationary

Section 1's worked trace picked one specific choice — hold the weight fixed in each PE, stream activations and partial sums past it — without stating that it was a choice. It was. The reduction a matmul performs, y_j = Σ_i w_ij · x_i, has three participants — weights, activations, and the partial sum accumulating the reduction — and a systolic array's defining design decision is which one of those three stays resident in a PE while the other two move. That decision is called the dataflow, and it determines, structurally, which operand's movement gets minimized and which two operands pay the cost of streaming.

3.1 Weight-stationary

This is the dataflow Section 1.2 traced. Each weight is loaded into its PE once and held for the entire duration that tile of weights is active; activations and partial sums move through the array every cycle. What gets minimized is weight traffic — specifically, weight reads from off-chip memory, which matters because in a typical convolution or fully-connected layer, a given weight is reused across every spatial position in an activation map (or, at the extreme, across an entire batch), so if that weight is resident for the whole time it's being reused, it's fetched from DRAM exactly once no matter how many activations stream past it.

Weight-stationary: weight fixed, activation + partial sum move
 
        w --- PE --- w --- PE
              |             |
        activation →  activation →
              |             |
           psum ↓        psum ↓

Weight-stationary is the right call when the reuse ratio favors it: a relatively small, fixed set of weights (a bounded number of output channels, a bounded kernel size) gets reused across a large number of activation vectors — large batch, long sequence, or large spatial extent relative to how much weight data there is. The TPU's MXU, as traced above, is the textbook production example of exactly this choice, made because Google's own workload characterization found datacenter inference dominated by large-batch matmuls where weight reuse across the batch was substantial (Jouppi et al., 2017).

3.2 Output-stationary

Here the partial-sum accumulator is what stays fixed in a PE for the duration of one output element's full reduction; weights and activations both stream through. What gets minimized is partial-sum read/write traffic — which matters more than it might look like at first, because partial sums are frequently carried at higher precision than the operands that produced them (a 32-bit accumulator summing 8-bit × 8-bit products, to avoid overflow across a long reduction), so every read-modify-write of a partial sum moves more bytes per element than moving the weight or activation that generated it.

Output-stationary: partial sum fixed, weight + activation move
 
        weight →  PE  ←  activation
                    |
                 psum (held, accumulating in place)

Output-stationary earns its keep when the reduction dimension is deep — a large number of input channels or a large kernel footprint being summed into one output. In that regime, keeping the accumulator resident for the entire reduction avoids many read-modify-write round trips per output element that a design streaming partial sums (like the weight-stationary array above) would otherwise pay.

3.3 Activation-stationary (input-stationary)

The mirror case of weight-stationary: a given activation value stays resident in a PE while weights and partial sums stream past it. What's minimized is activation movement.

Activation-stationary: activation fixed, weight + partial sum move
 
        activation --- PE ---
                        |
                   weight →
                        |
                     psum ↓

This is the right call in the opposite reuse regime from weight-stationary: when a single activation value gets reused against many different filters — a large output-channel count relative to a small spatial or batch extent, so one input value's contribution is computed against far more weights than any one weight is reused against inputs.

3.4 Choosing among them

Dataflow	What's fixed in the PE	What streams through	Traffic minimized	Favored when
Weight-stationary	Weight	Activations, partial sums	Weight reads from DRAM	Small, reused filter set streamed against a long activation sequence — large batch or long spatial extent relative to filter size
Output-stationary	Partial-sum accumulator	Weights, activations	Partial-sum read/write traffic	Deep reduction dimension (large `C_in`, large kernel) — accumulator would otherwise round-trip repeatedly
Activation-stationary	Activation	Weights, partial sums	Activation reads	Large output-channel count relative to spatial/batch extent — one activation reused against many filters

None of these three dominates the others in the abstract, which is precisely the point the source material's outline was making when it listed tensor dimensions, reuse patterns, SRAM size, bandwidth, and hardware topology as the deciding factors — the "best" dataflow is a property of the specific layer being executed, not a property of the accelerator alone, and a real NPU's dataflow is fixed at design time while the layers it has to run vary enormously in their reuse profile. This is also why the taxonomy above, while the classic three-way split, is not the end of the story: Chen, Emer, and Sze's Eyeriss work extended it to a fourth category, Row-Stationary, specifically because pure weight-, output-, or activation-stationary designs each leave some data type's reuse unexploited — Row-Stationary instead maps one filter row across a diagonal stripe of the PE array and schedules weight, activation, and partial-sum reuse simultaneously, minimizing total data movement energy across all three operand types rather than optimizing any single one in isolation (Chen, Emer, and Sze, ISCA 2016). It's a genuinely more sophisticated design than any single-stationary array, and it's the concrete proof that this taxonomy is a starting vocabulary for dataflow design, not a closed set of three options — worth knowing by name even though this lesson's worked examples stay with the three classic cases.

4. Compiler + Hardware Co-Design: Fitting a Graph to a Fixed Array

Every dataflow choice from Section 3 gets built into silicon once, at design time, for a fixed array size — an N×N grid of PEs that does not change shape after tape-out. A real model graph, meanwhile, has convolution and matmul layers of essentially arbitrary dimensions, chosen by whoever trained the model, with no obligation to match whatever N the target NPU happens to have. Reconciling those two facts — a fixed hardware shape and an arbitrary graph shape — is not a performance nicety layered on top of NPU compilation; it is most of what an NPU compiler's backend actually does, and it's the concrete answer to why "NPU compiler" is a distinct engineering discipline rather than a thin code-generation pass bolted onto a general ML compiler.

4.1 Tiling a conv layer onto a fixed PE grid

Take a hypothetical weight-stationary NPU with a 64×64 PE array, and a pointwise (1×1) convolution layer with C_in = 100 and C_out = 100 feeding a 56×56 spatial output — 3,136 spatial positions, each one a length-100 dot product against a 100-output filter bank, which is exactly a [3136, 100] × [100, 100] GEMM once the 1×1 kernel is written that way.

Neither 100 fits the array's 64 dimension, so the compiler has no choice but to tile the weight matrix into a 2×2 grid of sub-tiles along (C_in, C_out), with boundaries at 64 and the remaining 36:

Each of these four tiles is a separate weight-load into the array, followed by a full pass of all 3,136 spatial activation vectors streamed through before the next tile's weights can load — a weight-stationary array pays a real reload cost to swap tiles, so the compiler's schedule has to treat "how many distinct weight tiles does this layer decompose into" as a first-class cost, not an afterthought.

The utilization numbers matter because a partially filled tile does not run any faster than a full one — the array still cycles once per streamed activation regardless of how many of its PEs are actually holding a non-zero weight, so the unused PEs in a 64×36 tile are not idle in the sense of saving time, they're multiplying real activations against zero-padding, burning exactly as many cycles as a fully-utilized tile would, for zero useful arithmetic. Summing the real work against the provisioned capacity makes the waste concrete: the layer needs 100 × 100 = 10,000 weight elements of actual work, but the compiler had to provision 4 × 64 × 64 = 16,384 PE-cells' worth of tile capacity to cover it — a utilization of 10,000 / 16,384 ≈ 61%, meaning roughly 39% of this layer's PE-cycles on this array are spent computing against zero, purely because 100 isn't a multiple of 64.

Contrast that against the same layer with C_in = C_out = 128 — an exact multiple of the array's 64 dimension. Tiling becomes a clean 2×2 = 4 full tiles, zero padding, 100% utilization on every tile. This is the same argument Lesson 4 made about choosing channel counts that are multiples of a CPU's SIMD width (the C = 19 on 4-wide NEON example, where the non-vectorizable remainder cost a real fraction of total runtime), just one level coarser: instead of aligning to a 4- or 8-wide vector register, a model targeting a specific NPU benefits from having its channel counts land on multiples of that NPU's native PE-array dimension — 64, 128, or whatever the silicon actually implements — and this is a real, first-order reason production NPU-targeted architectures (and the padding decisions their compilers make automatically when they don't) look the way they do.

4.2 Loop order as a structural constraint, not just a tuning knob

Lesson 4's GEMM material treated loop order — ijk versus ikj versus any of the other four orderings around a triple-nested reduction — as a tuning space: every ordering computes the correct answer, and the choice between them is purely about which access pattern plays well with the cache hierarchy underneath. On a fixed-dataflow NPU, that freedom mostly disappears, because the hardware's physical wiring only supports specific movement directions for specific data.

For the weight-stationary array traced in Section 1, the two dimensions that get spatially unrolled onto the PE grid's rows and columns — here, C_in and C_out — have to be the outer, tile-defining loops, because those are the dimensions whose weight-load cost the compiler pays once per tile and wants to amortize over as much streamed work as possible. The spatial/batch dimension has to be the innermost loop, because it's the one axis that's actually streamed at one vector per cycle through an already-loaded array. A compiler that instead tried to make the spatial dimension the outer loop — reloading a fresh set of weights on every spatial step — wouldn't produce a slower-but-correct program the way a bad cache-blocking choice does on a CPU; it would produce a program bound entirely by weight-load bandwidth rather than MAC throughput, defeating essentially the entire reason the systolic array exists. In the worst case, an assignment that tries to spatially unroll the wrong pair of dimensions onto a fixed-wiring array isn't just slow — it isn't a legal instruction sequence for that hardware's ISA at all, and the compiler backend has to reject it outright rather than merely deprioritize it. Loop order on this hardware is closer to a correctness constraint dictated by the dataflow than a performance knob the compiler tunes.

4.3 Operator-support gaps and CPU fallback

Lesson 5's Part 2 covered the runtime's job of deciding which device executes which operator, and quantified the real synchronization cost of crossing between CPU and NPU mid-graph. This lesson can now explain why that decision has to be made at all, rather than treating "some ops just run better on CPU" as an empirical fact to be discovered by benchmarking: a systolic array's dataflow is fixed in silicon at design time, and it can only natively express the small family of computations its wiring was built to stream — dense, regular reductions that look like matmul or convolution. Operators with irregular structure — data-dependent control flow, scatter/gather, top-k or non-max-suppression, reshapes that don't respect the array's native tiling, or precisions the MAC datapath simply doesn't implement — either have no legal mapping onto the array's fixed wiring at all, or would require emulating them as such a degenerate sequence of matmul-shaped operations that the CPU fallback, synchronization cost and all, is still faster.

Seen this way, the operator-placement dispatch logic Lesson 5 Part 2 described is downstream of a decision made much earlier and much more permanently: at the point the NPU's dataflow and instruction set were designed, some fixed set of operator patterns was chosen as the set the silicon can express, and every operator outside that set is a CPU fallback by construction — not a gap the runtime discovers empirically at deployment time, but a gap that was determined the moment the array's wiring was fixed.

4.4 Co-design at instruction-selection granularity

The co-design story doesn't stop once an operator is correctly placed on the NPU and correctly tiled to the array — there's a finer-grained decision left at the level Lesson 7 called instruction selection. Some NPU instruction sets expose a fused INT8-dot-product instruction that consumes a short vector of INT8 operand pairs and accumulates their products in one issue, rather than requiring the datapath to be driven by one scalar multiply-accumulate per operand pair. If the compiler's lowering pass emits the naive sequence —

INT8 multiply
INT8 multiply
INT8 multiply
INT8 multiply

— issued as four separate operations where a single fused dot-product instruction was available, the array's peak throughput for that operator is left entirely on the table, even though every earlier decision (placement, tiling, dataflow, loop order) was made correctly. This is architecturally the same failure mode Lesson 7 described for a CPU compiler that fails to select a fused-multiply-add or a packed-SIMD instruction when the pattern was sitting right there in the IR — just applied to an NPU's native instruction set instead of LLVM's. It's the sharpest one-line summary of everything this section has been building toward: an NPU can only run as fast as the narrowest compiler decision mapping onto it, at every level from graph-level operator placement, through scratchpad tiling and dataflow-legal loop order, down to the literal instruction encoding at the bottom of the stack. Good hardware requires a compiler that maps onto it correctly at all of those levels simultaneously, and good inference software has to understand every one of those levels to write that mapping.

5. Everything At Once: The Full Optimization Pipeline, Stacked

Lesson 10 — and really, this entire series — has been building toward a single practical question: what happens when every optimization covered so far is applied to the same model, in sequence, rather than studied in isolation? The source material's own worked example makes this concrete with a small CNN (Conv → ReLU → Conv → BatchNorm → ReLU → GEMM), starting from a deliberately unoptimized baseline and applying six changes, each one traceable to a specific earlier lesson in this series:

Stage	Optimization	Where it was covered	Latency
0	Baseline: FP32, CPU, dynamic allocation, DRAM intermediates, no fusion, generic kernels	—	100 ms
1	INT8 quantization	Lesson 3 — affine/symmetric quantization	70 ms
2	Operator fusion (`Conv + ReLU` → one kernel)	Lesson 2 — graph-level optimization	60 ms
3	Memory planning (activation buffer reuse)	Lesson 5, Part 1 — memory planning	55 ms
4	Optimized, vectorized GEMM	Lesson 4 — kernel engineering	35 ms
5	NPU offload	Lesson 10, Parts 1 and 3 — this lesson	10 ms
6	Eliminated CPU/NPU transfers	Lesson 5, Part 2 — runtime scheduling	7 ms

Each step is doing something this series has already derived from first principles, not applying a new trick. INT8 quantization (100ms → 70ms) shrinks every weight and activation to a quarter of its FP32 footprint and lets the hardware issue integer arithmetic instead of floating point, exactly the scale-and-zero-point mechanics Lesson 3 worked through numerically. Fusing Conv and ReLU into one kernel (70ms → 60ms) eliminates a full round-trip of the intermediate activation tensor through memory and a second kernel-launch's worth of overhead, the same fusion argument Lesson 2 made from the roofline model. Memory planning (60ms → 55ms) removes dynamic allocator overhead and improves locality by reusing a small set of pre-planned buffers instead of allocating fresh ones per tensor, Lesson 5's graph-coloring buffer-reuse argument. Swapping a generic GEMM for a properly blocked, vectorized one (55ms → 35ms) is Lesson 4's entire register-blocking derivation paying off directly, moving the kernel from memory-bound toward the compute roofline. NPU offload (35ms → 10ms) is this lesson's contribution specifically: the same GEMM, now executed on a systolic array running the reduction spatially across a PE grid instead of temporally on a CPU's ALUs, with a dataflow chosen for the layer's actual reuse pattern and a compiler that tiled it correctly onto the array. And eliminating the CPU/NPU transfers (10ms → 7ms) is Lesson 5 Part 2's synchronization-cost argument applied in reverse — instead of measuring the tax of crossing devices, this step removes crossings that turned out to be unnecessary once the graph was restructured to keep more of the pipeline resident on one side.

Stacked together, 100ms → 7ms is roughly a 14.3× speedup, and the thing worth sitting with is where that speedup came from. No architectural search happened. No layer was added, removed, or resized. The network computes the exact same function at the end as it did at the start — every one of these six numbers came from changing how the same graph gets executed: what precision it's computed in, how its operators are grouped, where its intermediates live, what kernel implements its inner loop, which piece of silicon runs which operator, and how much synchronization crosses between them. That is the entire discipline this series has been building toward one lesson at a time, and Lesson 10 closes on an NPU specifically because it is the deployment target where every one of those decisions — precision, fusion, memory placement, kernel selection, and now dataflow and tiling — has to be gotten right simultaneously, on hardware with essentially no tolerance for getting any one of them wrong.