The Compiler's Role: IR Levels, Kernel Selection, and Layout Transformation

Lesson 6 spent its energy on transformers in production — attention, KV caches, continuous batching, all the serving-level machinery that keeps a model fed with requests. Every bit of that discussion quietly assumed something: that when an attention layer's matmul hit the hardware, some correct, reasonably fast kernel already existed to run it. Lesson 7 is about the thing that decides which kernel that is, and it turns out to be one of the least well-understood pieces of the entire inference stack: the compiler.

1. What "The Compiler" Actually Owns Here

The ISA-vs-microarchitecture piece from the companion series already walked the general compiler pipeline this reader knows professionally — AST, IR, optimization passes, instruction selection, register allocation, down to bits. That pipeline is real and it still runs, somewhere, underneath every ML compiler. But it is not sufficient on its own, and the reason is specific to this domain.

A C++ compiler's job starts from source that is already close to what the machine will eventually do: a for loop in C++ is, modulo optimization, a for loop on the hardware. An ML compiler's job starts from something much further from the machine: a PyTorch graph node called Conv2d doesn't describe a loop at all. It describes an operator — a named mathematical transformation with a well-known reference semantics, but no committed loop order, no committed memory layout, no committed algorithm, and often not even committed shapes until the graph is actually traced or shape-inferred. The distance between "Conv2d(x, w, stride=1)" and "the sequence of vector-FMA instructions that actually execute on this NPU's MAC array" is enormously larger than the distance between a C++ for loop and its assembly, and that distance is exactly what an ML compiler exists to close.

Everything in this lesson is about how that distance gets closed — not in one leap, but through a stack of intermediate representations, each one solving a problem the levels above and below it cannot solve on their own.

2. Why One IR Cannot Do This Job

Start from the constraint, stated precisely, because the rest of the lesson is really just this one idea worked out in detail: a single IR cannot simultaneously be close to a source framework's semantics and close to target hardware's execution model, because the properties that make an IR good for one job actively work against the other job.

Consider what an IR needs to expose to make graph-level decisions cheap — operator fusion, algebraic simplification, constant folding, the whole toolkit from Lesson 2. Those passes want to reason about a small number of large, semantically named nodes: "this is a convolution, it's followed by a batch-norm, which is followed by a relu — I recognize this pattern, I can fold it into one node." That recognition is fast and reliable because each node still carries its high-level identity. The pass never has to prove, from scratch, that a thousand scalar instructions happen to implement a convolution; it's told so directly by the IR.

Now consider what an IR needs to expose to make kernel-level decisions cheap — vectorization, register allocation, instruction scheduling, the toolkit from Lesson 4. Those passes want the opposite: an explicit loop nest with concrete induction variables, concrete trip counts (or at least provably-affine bounds), and concrete memory accesses, because vectorization has to reason about which iterations touch which addresses and whether that access pattern is contiguous enough to pack into a vector register. A node that just says Conv2d gives a vectorizer nothing to work with — there's no loop to vectorize yet.

These two requirements are in direct tension. If you force the graph-level IR to always look like an explicit loop nest, fusion becomes an enormous, brittle pattern-matching problem over thousands of individual instructions instead of a small rewrite over a handful of named nodes — you've thrown away the very information ("this is one conv") that made the fusion decision cheap. If you force the kernel-level IR to stay at the Conv2d-node level of abstraction, no vectorizer or register allocator can do its job, because there's no loop, no address arithmetic, and no register pressure to reason about yet — those things only exist once the operator has been expanded into an actual computation.

The resolution the field converged on is not a clever single IR that avoids this tradeoff. It's the admission that the tradeoff is real, and the fix is to use several IRs, each one matched to the passes that want to run at that level, connected by well-defined lowering steps that translate meaning downward without losing it. That's what "multi-level compilation" means, and it's worth being precise that this isn't unique to ML — LLVM already has this shape internally (Clang's AST, LLVM IR, target-specific SelectionDAG, MachineInstr, and finally real machine code, as the companion post covered). What's specific to ML compilers is how much more abstraction distance has to be covered, and MLIR is the infrastructure the field built specifically because that distance was too large for LLVM IR alone to span.

3. MLIR's Dialect Stack as the Concrete Case Study

MLIR's core design decision is that instead of one fixed intermediate representation, it defines a dialect system: a dialect is a self-contained namespace of operations, types, and attributes that models one abstraction level well, and MLIR's infrastructure (the pass manager, the rewrite framework, the verifier) is generic over which dialect a given piece of IR happens to be written in. A program moves through several dialects over the course of compilation, getting progressively lowered — rewritten from a higher-abstraction dialect into a lower-abstraction one — until it reaches a dialect close enough to real hardware that code generation is a mechanical, almost boring step.

A representative pipeline, close to what a real ML compiler stack (TensorFlow's IREE, Torch-MLIR-based backends, and this reader's own NPU compiler domain all follow some variant of) actually does, looks like this:

PyTorch / TensorFlow / ONNX graph
            │  (frontend import — framework-specific dialect:
            │   torch, tf, or an ONNX-derived representation)
            ▼
   Framework dialect  (op names, dynamic shapes, framework-specific quirks)
            │  (legalization to a common operator set)
            ▼
   TOSA / Linalg-on-Tensor  (named, hardware-agnostic ops: conv2d, matmul, generic)
            │  (bufferization: tensor value-semantics → memref side-effecting memory)
            │  (tiling, fusion, layout selection happen around here)
            ▼
   Linalg-on-Memref / Affine  (explicit loop nests, affine bounds and accesses)
            │  (loop transformations: tiling, unrolling, interchange, vectorization)
            ▼
   SCF + Vector dialect  (structured control flow, explicit fixed-width vector ops)
            │  (lowering to LLVM's type system and instruction set)
            ▼
   LLVM dialect  (MLIR's own mirror of LLVM IR)
            │  (translation out of MLIR entirely)
            ▼
   LLVM IR
            │  (instruction selection, register allocation — covered in the
            │   companion ISA-vs-microarchitecture post)
            ▼
   Machine code

This is not the only valid pipeline — a GPU backend swaps the tail end for the SPIR-V dialect, and an NPU compiler typically inserts one or more additional hardware-specific dialects between Linalg and the final code-generation step to model a MAC array, DMA engines, or scratchpad memory that has no LLVM-level analogue at all. But the shape of the pipeline — several dialects, each closer to hardware than the last, connected by explicit, verifiable lowering passes — is the general pattern, and it's worth walking through what problem each transition specifically solves.

3.1 Frontend dialect → TOSA / Linalg-on-Tensor

The first real transition matters because it's where framework-specific baggage gets stripped away. A PyTorch graph's torch dialect still carries PyTorch's op set, PyTorch's default type inference quirks, and often dynamic shapes that haven't been resolved yet. TOSA — the Tensor Operator Set Architecture — exists precisely to give every frontend a common landing spot: it was developed by reconciling a "top-down" view (what operators do real frameworks actually emit, in practice, across dozens of real networks) with a "bottom-up" view (what operator granularity do real CPU, GPU, and NPU backends actually want to consume), and it currently covers on the order of a hundred operators chosen from occurrence-frequency data across real TensorFlow and TensorFlow Lite models, spanning both floating-point and quantized int8 content. Once a graph is in TOSA (or the closely related Linalg-on-Tensor form used by many other pipelines), every downstream pass — fusion, layout selection, quantization — gets to be written once, against one operator vocabulary, instead of once per frontend.

Linalg deserves its own mention here because it plays two roles in this stack, not one. At the tensor level, Linalg's "named" ops (linalg.matmul, linalg.conv_2d, and so on) behave like TOSA: high-level, hardware-agnostic, fusion-friendly nodes. But Linalg's defining trick is that every one of those named ops is really syntactic sugar over a single, more general primitive — linalg.generic — which explicitly declares its iteration space, which dimensions are parallel versus reduction, and how tensor indices map to loop indices. That declaration is what lets a Linalg op "know how to decompose itself into loops" on demand, mechanically, without a separate hand-written lowering rule per operator. It's the hinge the whole graph-to-loop transition swings on.

3.2 Bufferization: tensors become memory

A subtlety worth calling out explicitly, because it's easy to skate past and it matters for anyone who has actually written an MLIR pass: up through the Linalg-on-Tensor level, MLIR's tensors have value semantics — a tensor value is immutable, like an SSA value, and an op that "produces a new tensor" doesn't imply any particular memory write happened yet. Bufferization is the pass that decides, for the first time, where each tensor actually lives — which ones get a physical buffer (a memref), which ones can alias an existing buffer in place, and which ones can be eliminated because nothing downstream ever reads them (dead-tensor elimination, the MLIR-level cousin of Lesson 2's dead-node elimination). This is also, not coincidentally, close to where the memory-planning decisions from Lesson 5 — arena allocation, in-place execution, buffer reuse — get made concrete. Before bufferization, "does this op need a scratch buffer" isn't even a well-formed question, because nothing has memory yet.

3.3 Linalg-on-Memref / Affine: the loop nest appears

Once tensors are buffers, Linalg ops can be lowered to actual loops — and this is the transition where "loop-level IR" stops being a metaphor. The Affine dialect specifically restricts loop bounds and memory-access index expressions to affine functions of loop induction variables and symbols (linear combinations, no arbitrary control flow inside the access expression). That restriction looks limiting, but it's the entire point: affine structure is precisely what lets the compiler apply polyhedral-style loop transformations — tiling, fusion, interchange, skewing — with mathematical guarantees about correctness, the same way a human kernel engineer reasons about a nest of for loops when hand-blocking a GEMM. This is the dialect level where Lesson 2's tiling derivation and Lesson 4's BLIS-style blocking hierarchy stop being things a human writes by hand and become things a compiler pass can derive and apply automatically, provided the loop nest is affine enough for the pass to trust its own bound and dependence analysis.

3.4 SCF, Vector, and the LLVM dialect

Not every loop nest that shows up in a real model is affine — dynamic shapes, data-dependent control flow, and some fused patterns fall outside what the Affine dialect can represent, and those get expressed instead in SCF (structured control flow: ordinary for and while loops without the affine restriction). The Vector dialect is where SIMD becomes explicit and target-independent: a loop that's been tiled down to, say, a strip of 8 elements gets rewritten into operations on an 8-wide vector type, still without committing to whether that vector maps to AVX2, NEON, or an NPU's native vector width — that final commitment happens only when Vector-dialect ops lower into the LLVM dialect, MLIR's own in-house mirror of LLVM IR, which is then translated out of MLIR entirely into real LLVM IR and handed to the LLVM backend this reader already knows from the companion post: instruction selection, register allocation, down to machine code.

3.5 Why progressive lowering, specifically, beats one giant IR

Put the whole stack together and the design argument becomes concrete rather than abstract. Every one of the transitions above deletes information that its own level didn't need and would have gotten in the way of ("this was originally a PyTorch nn.Conv2d call" is irrelevant noise by the time you're vectorizing a loop) while adding information the next level specifically requires ("this loop's bounds are affine, its access pattern is stride-1 in the innermost dimension" is exactly what a vectorizer needs and a graph-fusion pass has no use for). A single IR trying to carry both kinds of information simultaneously, for the entire compilation, would either bloat every op with fields nobody after the first three passes ever reads, or force every pass to reconstruct high-level structure from low-level detail it was never given cleanly in the first place — which is precisely the "prove from scratch that a thousand instructions implement a conv" problem Section 2 raised. Progressive lowering through a dialect stack is what lets each pass operate at exactly the abstraction level its analysis needs, and lets the framework guarantee — via each lowering's own correctness proof, checked once — that meaning is preserved on the way down.

4. Graph IR vs. Kernel IR, Worked Concretely

With the stack in view, it's worth making the graph-IR-vs-kernel-IR distinction as concrete as the source material's own example, because the gap between these two representations of the same computation is the entire reason Section 3's machinery exists.

At the graph-IR level (TOSA or Linalg-on-Tensor), a small network fragment looks like this:

%1 = tosa.conv2d(%input, %weights)  { stride = [1, 1] }
%2 = tosa.relu(%1)
%3 = tosa.matmul(%2, %weights2)

Three nodes. No loops, no addresses, no registers — just named operators and their data dependencies. This is exactly the representation a fusion pass wants: it can look at %1 and %2, recognize a conv-then-relu pattern, and decide to fuse them into one op without ever caring what a "loop" is.

At the kernel-IR level, after that same tosa.conv2d has been lowered through Linalg to Affine, its literal semantics — for a direct-convolution lowering, before any algorithm substitution — look like this:

for n  in 0 .. N          // batch
  for co in 0 .. C_out     // output channel
    for ho in 0 .. H_out   // output row
      for wo in 0 .. W_out // output column
        acc = 0
        for ci in 0 .. C_in   // input channel   (reduction)
          for kh in 0 .. KH   // kernel row       (reduction)
            for kw in 0 .. KW // kernel column    (reduction)
              hi = ho * stride_h + kh
              wi = wo * stride_w + kw
              acc += input[n, ci, hi, wi] * weight[co, ci, kh, kw]
        output[n, co, ho, wo] = acc

Seven nested loops, explicit induction variables, explicit index arithmetic, an explicit accumulator. This is exactly the representation a vectorizer, a register allocator, and a loop-tiling pass want, and exactly the representation Lesson 4's whole toolkit — loop reordering, blocking, register-tile sizing, SIMD lane packing — operates on. Nobody hand-writes this loop nest for every model; it's what tosa.conv2d (or linalg.conv_2d) decomposes into, mechanically, the moment the compiler needs to reason about memory access and arithmetic instead of dataflow.

Now notice something the two representations expose asymmetrically: the graph-IR node says nothing about how the reduction over ci, kh, kw gets computed — only that it's a convolution. The kernel-IR loop nest above is the literal, direct-convolution reading of that reduction, but it is emphatically not the only correct lowering. Lesson 4 already established that a fully-connected layer and a convolution both reduce to the same underlying primitive — a dot product summed into an output element — and that the highest-leverage way to execute that primitive on real hardware is GEMM, not a hand-rolled seven-deep loop nest. The kernel-IR level is exactly where that choice gets made: the compiler can lower tosa.conv2d either into the direct loop nest shown above, or into an im2col-plus-GEMM formulation — a pack loop that materializes (or, in the implicit-GEMM case covered in Lesson 4, virtually materializes) the unfolded input matrix, followed by the same BLIS-style register-blocked GEMM micro-kernel that every fully-connected layer in the network already uses. Both lowerings are semantically identical loop nests over the same mathematics; they differ enormously in which one a real vectorizer and register allocator can actually turn into fast code, because GEMM's loop structure is exactly the one Lesson 4 showed can be blocked into 170+ FLOP/byte arithmetic intensity, while the naive seven-loop nest above, left untiled, re-reads input and weight from memory on nearly every iteration. This is not a graph-IR decision — the graph IR doesn't even have a concept of "loop nest" to choose between — it's a kernel-IR decision, made at exactly the abstraction level that can see loops and memory traffic at all. It is also, not coincidentally, the same decision Section 6 below revisits as kernel selection: im2col-plus-GEMM versus direct convolution versus Winograd are three different kernel-IR-level implementations of one graph-IR-level node.

5. Layout Transformation as a Genuine Compiler Pass

Lesson 2 already established why NCHW and NHWC exist and why hardware backends disagree about which one they want fed to them — channel-vectorized NPU and DSP MAC arrays want channels contiguous (NHWC), GPU-lineage kernels historically want each channel's spatial plane contiguous (NCHW). What this lesson adds is where in the dialect stack that decision actually has to be made, and why getting that placement wrong is itself a correctness-adjacent compiler bug, not just a missed optimization.

Layout is fundamentally a graph-IR-level concept. At the TOSA/Linalg-on-Tensor level, a tensor's layout is metadata — an attribute or a type annotation saying "this 4D tensor's dimensions are ordered N, C, H, W" — attached to a value that otherwise has no concrete memory address yet. That's exactly the representation a layout-optimization pass needs: it can look at an entire subgraph, see which ops prefer which layout, and decide where transforms are unavoidable, all without touching a single loop or address computation. Once the graph has been bufferized and lowered to Affine loops (Section 3.3), that decision window is gone — layout has been baked directly into which memory address each loop iteration computes into, and by that point "insert a layout transform here" is no longer a clean, localized graph edit; it's a full loop-nest rewrite entangled with whatever tiling and vectorization decisions have already been applied. This is the concrete, mechanical reason layout selection has to happen at the graph-IR level of the stack, not the kernel-IR level — it's not a style preference, it's that the information needed to make the decision cheaply stops existing once the tensor becomes a buffer.

5.1 A minimal example with a real, unavoidable transform

Take a tiny fragment: an NPU-targeted depthwise convolution (wants NHWC, because the vector unit reduces across the channel axis) feeding into a pointwise convolution implemented as an im2col-plus-GEMM kernel tuned against a GPU-lineage code path (wants NCHW, because that's the layout its packing routine was written against). The compiler cannot silently reconcile this — the two kernels genuinely expect data laid out differently in memory, and running one on the other's layout without a transform would just be reading the wrong bytes as the wrong tensor elements. Something has to insert an explicit layout-conversion op:

%1 = depthwise_conv2d(%input)         : tensor<NHWC>
                    │
                    ▼   (layouts mismatch: producer is NHWC, consumer wants NCHW)
%1t = transpose(%1, perm=[0,3,1,2])   : tensor<NHWC> → tensor<NCHW>
                    │
                    ▼
%2 = pointwise_conv2d(%1t)            : tensor<NCHW>

That transpose is a real op with a real cost: it's a memory-bound pass that reads the entire activation tensor in one stride order and rewrites it in another, touching every byte with no arithmetic to amortize the traffic against — Lesson 2 already quantified this as potentially rivaling the cost of the cheaper convolution it sits next to. The naive compiler strategy — insert a transform at every point in the graph where a producer's layout doesn't match a consumer's preference — is correct but can be badly wasteful, because it treats each mismatched edge independently instead of recognizing that many of those transforms cancel out.

5.2 Minimizing the count, not just avoiding wrong answers

The fix is layout propagation, and it's worth stating precisely why it works. Most operators in a real network — activation functions, elementwise add, batch-norm's scale-and-shift, even a stride-1 same-padding convolution's own internal computation — are layout-agnostic in the sense that they don't care which axis order their input arrives in; they just need to apply the same operation independently at every spatial-and-channel position. A layout-optimization pass exploits that by picking one preferred layout and propagating it through every layout-agnostic op in a chain, converting only at the genuine boundaries — points where an op has a hard, layout-specific requirement, like the depthwise-then-pointwise example above, or the network's own input/output tensors, which usually arrive in whatever layout the framework or the data pipeline produced them in.

Concretely, for a chain like conv → relu → conv → relu → conv where every conv genuinely prefers NHWC and relu is layout-agnostic, the naive per-edge strategy would insert zero transforms here regardless (all preferences already agree) — but extend the chain to conv(NHWC) → relu → conv(NCHW-only kernel) → relu → conv(NHWC) and the propagation pass reduces what could naively look like several transform sites down to exactly two: one entering the NCHW-only stretch, one leaving it, rather than a transform bracketing that single op on both sides regardless of what's upstream and downstream. TVM's Relay-level ConvertLayout pass and ONNX Runtime's NCHWc layout optimizer, both already cited in Lesson 2, are production instances of exactly this propagate-then-convert-only-at-boundaries strategy, and it's a strategy that is only expressible at all because the graph IR still has "layout" as an explicit, first-class, reasoned-about property — the same structural argument Section 3 made about fusion, now applied to a different pass.

6. Kernel Selection as a Decision Problem

Once an operator's layout is fixed and its shapes are concrete, the compiler still hasn't picked which code runs. A single logical operator — convolution, matmul, even something as simple as elementwise add — typically has several independently-implemented kernel candidates, each with genuinely different performance characteristics depending on hardware, shape, datatype, and layout:

Picking the right one is a real decision problem, not a lookup — the "best" kernel for a 3×3, stride-1, 64-channel, batch-1, int8 convolution on one CPU can be measurably worse than the second-best kernel for a 1×1, stride-2, 512-channel, batch-8, fp32 convolution on the same CPU, because the two shapes stress completely different resources (arithmetic intensity, register pressure, vectorization-friendliness, memory footprint) in completely different proportions. This is why every serious inference engine — TVM, XNNPACK, oneDNN, TensorRT, and NPU-vendor compiler stacks alike — has some form of dispatch logic: code that looks at the concrete (op, shape, dtype, layout, hardware) tuple at compile time (or sometimes at first-run time) and routes to the kernel implementation that tuple predicts will win.

There are two fundamentally different ways to build that dispatch logic, and the tradeoff between them is one of the most consequential engineering decisions an inference-compiler team makes.

6.1 Autotuning: search the space, measure on real hardware

AutoTVM, from the "Learning to Optimize Tensor Programs" line of work, treats kernel selection as a search problem over a schedule template: a human writes an abstract description of a kernel (say, a tiled GEMM) parameterized by tunable knobs — tile sizes along each dimension, unroll factors, which axis to vectorize, loop order — and AutoTVM searches that parameter space, compiling and actually running candidate configurations on the target hardware, using a learned cost model (originally gradient-boosted trees, later a TreeGRU-style model) to prioritize which configurations are worth measuring next rather than exhaustively trying all of them. The result is a schedule tuned specifically for one (operator, shape, hardware) triple, often competitive with hand-written vendor libraries for that exact triple — but it depends entirely on a human having written a good template in the first place, and a template that's good for GEMM says nothing about depthwise convolution.

Ansor (TVM's auto-scheduler) removes that dependency. Instead of requiring a hand-written template, Ansor derives its own search space algorithmically straight from the computation's definition — generating structural "sketches" (candidate high-level transformation skeletons: which loops to tile, where to place a cache stage) and then filling in the concrete tunable details via random sampling and evolutionary search, still guided by a learned cost model and still validated against real hardware measurements. It also optimizes across whole subgraphs rather than one operator template at a time, which lets it discover fusion-and-schedule combinations a human template author would be unlikely to write by hand. The tradeoff Ansor accepts for that automation is the same one AutoTVM accepts, just amplified: real hardware measurement is not free, and searching a much larger, template-free space means more candidate programs need to be compiled and timed before the search converges.

6.2 Heuristics: a fast, hand-derived lookup

The alternative is a heuristic dispatch table — hand-written rules, informed by domain knowledge and offline benchmarking, that map a shape-and-hardware signature directly to a kernel choice without any search at compile time: "if this is a 1×1 convolution with C_in and C_out both multiples of 8, route to the pointwise-specialized GEMM path; if the kernel is 3×3 with stride 1 and C_in is small, route to the Winograd path unless the datatype is int8, in which case Winograd's numerical error is unacceptable, so fall back to im2col-plus-GEMM." This is exactly the kind of dispatch logic real production inference engines ship for the overwhelming majority of shapes they see in practice, because it costs essentially nothing at compile time and its behavior is auditable by a human reading the rule.

6.3 The tradeoff, stated plainly

Neither approach dominates the other, and production systems very often use both: an autotuned schedule as the primary path for the shapes that were part of the offline tuning run, with a heuristic fallback for shapes the tuning pass never saw — a new batch size at serving time, say, that the offline autotuning campaign didn't cover. The honest summary is that autotuning trades compile-time cost for near-optimal, shape-specific runtime performance, while heuristics trade some amount of left-on-the-table performance for compile-time cost that's effectively zero and behavior a human can reason about without running a search.

7. Specialized Kernels: What "Knowing the Constants" Actually Buys You

The last piece of the source material's outline — specialized kernels beating generic ones — is really asking a sharper question than it looks: why, mechanically, does knowing 3×3, stride=1, int8, C=64, batch=1 in advance let an implementation outperform a kernel that handles arbitrary kernel sizes, strides, dtypes, channel counts, and batch sizes? The answer is the same register-blocking and SIMD material from Lesson 4, but it's worth tracing exactly where the extra performance comes from, because "specialized code is faster" on its own is a slogan, not an explanation.

7.1 Where a generic kernel actually loses time

A truly generic N-dimensional convolution loop — the kind that has to correctly handle any kh, kw, any stride, any channel count, any dtype, because it's the one fallback path every other kernel in the dispatch table exists to avoid — pays several costs a specialized kernel doesn't:

Address arithmetic that can't be constant-folded. Recall the direct-convolution loop nest from Section 4. Every innermost-loop iteration computes input[n, ci, hi, wi] and weight[co, ci, kh, kw], and in row-major storage those index expressions expand into linear-offset computations like ((n*C_in + ci)*H_in + hi)*W_in + wi — three multiplies and three adds per tensor access, two tensor accesses per MAC, so up to six multiplies and six adds of pure address bookkeeping surrounding one multiply-add of actual arithmetic, in the worst case where none of it can be hoisted. A good compiler applies strength reduction and loop-invariant code motion to turn most of that per-iteration multiplication into an accumulating add of a fixed stride — but only to the extent the loop bounds are provably fixed at compile time. When kh, kw, stride, and C_in are runtime parameters, because the same generic kernel body has to serve every possible convolution shape the model might contain, the compiler frequently cannot prove enough about the bounds to fully strength-reduce, unroll, or vectorize the loop — it has to leave real index-multiplication instructions in the hot path, competing for the same issue ports and ALU cycles as the FMA that's doing the model's actual work.

Failed or partial vectorization. SIMD, from Lesson 4, only pays off when the compiler can prove the loop trip count and access pattern support it — ideally a compile-time-known multiple of the vector width, contiguous stride-1 access along the vectorized axis. A generic kernel with a runtime channel count can't guarantee C mod lane_width == 0, so it either emits a scalar fallback for the entire loop to stay safe, or emits a vectorized main loop plus a scalar remainder loop and pays the non-vectorizable-tail cost Lesson 4 already quantified — for small, common channel counts (Lesson 4's own example used C = 19 on 4-wide NEON), that tail can be a double-digit percentage of total loop trips running at effectively scalar throughput.

No fixed register-tile shape. BLIS-style GEMM performance, from Lesson 4's blocking-hierarchy derivation, comes from a register-resident micro-kernel sized exactly to the hardware's register file — a fixed MR × NR outer-product tile that stays in registers for the entire KC-deep reduction. A generic kernel that has to handle arbitrary reduction depths and arbitrary output tile shapes cannot commit to one fixed register-tile size at compile time; at best it picks a size and pads or loops around the remainder, losing some of the register-residency guarantee that made the specialized micro-kernel fast in the first place.

7.2 Where a specialized kernel spends that budget instead

A kernel written — or compiler-generated — specifically for 3×3, stride-1, int8, C_in = 64, batch = 1 gets to treat every one of those numbers as a compile-time constant, not a runtime value, and that changes what's possible at every layer discussed above. The 9-element (3×3) kernel-window reduction can be fully unrolled rather than looped, which both removes loop-overhead instructions and exposes every one of those 9 multiply-adds to the scheduler as independent, reorderable work. Every index expression involving kh, kw, or the fixed channel count collapses to a constant offset the assembler bakes directly into the load instruction's immediate field — the six-multiply address-arithmetic worst case from Section 7.1 becomes zero multiplies, because there's nothing left to compute; every address is known at compile time up to the two genuinely runtime values (which output pixel, which batch element). The register tile can be sized exactly to the ISA's vector width times whatever MR × NR the target's register file supports, with the same confidence a hand-tuned BLIS micro-kernel has, because C_in = 64 is known to be an exact multiple of common SIMD widths (64 / 32 = 2 full int8 AVX2 registers, 64 / 16 = 4 full NEON registers with SDOT), so there's no non-vectorizable tail to pay for at all.

None of this is exotic — it's the direct, mechanical consequence of deleting every place in the generic kernel where a runtime value forced the compiler to stay conservative. It's also exactly why Winograd convolution belongs in this section's story rather than only in the kernel-selection dispatch table above: Winograd's F(2×2, 3×3) transform reduces a 3×3 stride-1 convolution's multiply count algebraically (roughly a 2.25× to 4× reduction in raw multiplications, depending on the specific Winograd variant, at the cost of extra additions in the input and output transforms) — but that transform's numerical stability degrades badly enough under int8 quantization that it's typically restricted to floating-point or higher-precision paths, which is precisely the kind of shape-and-dtype-specific tradeoff a dispatch table, not a generic kernel, is supposed to encode. Specialized kernels win not because someone wrote cleverer assembly in the abstract, but because fixing the shape and dtype in advance deletes runtime uncertainty at every one of the layers — addressing, vectorization, register tiling, even algorithm choice — where a generic kernel is forced to hedge.

Quick Reference

Further Reading

• MLIR Rationale — Why MLIR? — the official design rationale for MLIR's dialect system and progressive-lowering philosophy.
• The 'linalg' Dialect Rationale — MLIR — the case for structured, named operations that decompose into loops on demand, and how Linalg fits between graph-level and loop-level IR.
• Tensor Operator Set Architecture (TOSA) Dialect — MLIR — the real operator set reconciling framework-level and hardware-level requirements that this lesson used as the graph-IR case study.
• Zheng, L. et al., "Ansor: Generating High-Performance Tensor Programs for Deep Learning" — the OSDI paper on template-free auto-scheduling, cited directly in Section 6.
• Apache TVM, "Introducing TVM Auto-scheduler (a.k.a. Ansor)" — the project's own accessible walkthrough of how Ansor's search differs from AutoTVM's template-based approach.
• "Introduction to Intermediate Representation (IR)" — GeeksforGeeks — a general grounding in why IRs exist and how high-level/low-level IR splits generalize beyond ML compilers specifically.

Lesson 8 turns from how the compiler executes a fixed model efficiently to shrinking the model itself: pruning, distillation, and sparse inference, and where each of those techniques' savings actually shows up in the kernels and IR levels this lesson just walked through.