Compute-Bound, Memory-Bound, and the Roofline Model

Lesson 1, Part 1 established that computation is not the whole problem — that counting FLOPs tells you almost nothing about wall-clock latency until you know how those FLOPs are actually fed. Part 2 turns that observation into a precise, quantitative tool: the Roofline model, and the single number — arithmetic intensity — that determines which side of the performance problem you are actually fighting.

1. Two Regimes: Compute-Bound and Memory-Bound

Every kernel that runs on real hardware spends its time doing exactly one of two things at any given moment: performing arithmetic, or waiting for operands to arrive. An inference engineer's first diagnostic question, before touching a single instruction, is which of these dominates. Get this wrong and you can spend a sprint hand-tuning an inner loop's FMA scheduling on a kernel that was never going to run faster than the DRAM bus allows — or, just as wastefully, add a fancier prefetcher to a kernel that was already saturating every ALU in the array.

1.1 Compute-bound: the arithmetic units are the bottleneck

A workload is compute-bound when, for its entire execution, the processor's multiply-accumulate (MAC) units, SIMD lanes, or tensor cores are continuously busy, and the memory system has no trouble keeping them fed. In this regime:

Time ──────────────────────────────────────────►
 
Compute units:  [MAC][MAC][MAC][MAC][MAC][MAC][MAC][MAC]   ← never idle
Memory bus:     [fetch][fetch] ... [fetch]                 ← finishes early, waits

The defining symptom is that adding more memory bandwidth does essentially nothing to latency, because the bandwidth was never the constraint — the arithmetic pipeline was. The only levers that move the needle are ones that touch computation directly: more parallel MAC units, higher clock frequency, better instruction scheduling to hide pipeline latency, or an algorithmic change that does strictly fewer FLOPs for the same result (e.g. Winograd convolution, or int8 quantization that halves the cycles per MAC on hardware with native low-precision support).

1.2 Memory-bound: the arithmetic units are starving

A workload is memory-bound when the compute units spend a meaningful fraction of their time idle, stalled on a load or store that hasn't completed yet:

Time ──────────────────────────────────────────►
 
Compute units:  [MAC]....[MAC]........[MAC]....[MAC]       ← frequent idle gaps
Memory bus:     [fetch][fetch][fetch][fetch][fetch][fetch] ← continuously busy

Here the arithmetic is, in a very real sense, free — the processor could do far more of it in the time available — but the memory system cannot supply operands fast enough to keep the pipeline full. Typical root causes:

• poor spatial or temporal locality (the access pattern doesn't reuse what's already close to the compute unit)
• tensors too large to fit in on-chip SRAM or cache, forcing repeated DRAM round-trips
• unfused operator chains that write and re-read intermediate results through DRAM
• unfavorable tensor layout (e.g. channel-last data fed to a kernel written assuming channel-first, forcing strided, non-coalesced accesses)
• low arithmetic intensity intrinsic to the operator itself, no matter how well it's implemented

In this regime, adding more ALUs is the wasted investment — you'd be building a bigger engine for a car that's stuck in traffic. The fix has to attack data movement: tiling, operator fusion, better layout, quantization to shrink bytes moved, or exploiting a faster tier of the memory hierarchy.

1.3 Two ML examples that make the distinction concrete

The cleanest illustration of "compute-bound" in deep learning inference is a large, well-tiled dense matrix multiply (GEMM) — the operation underlying every fully-connected layer and, via im2col or implicit-GEMM formulations, most convolutions. A large GEMM has O(N^3) arithmetic over only O(N^2) data, and a competent tiling scheme can reuse each loaded element many times before it leaves fast on-chip memory. Done well, this pushes the vast majority of execution time into the MAC array itself, which is exactly what an NPU's silicon budget was built to reward.

The cleanest illustration of "memory-bound" is the opposite extreme: an unfused chain of elementwise operations — think a naive implementation of Conv → BatchNorm → Activation where each stage is its own kernel launch that reads its input tensor from DRAM and writes its output tensor back to DRAM before the next stage starts. The arithmetic per element (a multiply-add for the batchnorm affine transform, a comparison-and-select for a ReLU) is trivially cheap; the intermediate tensor traffic dominates completely. A small-batch depthwise convolution is a second, subtler example of the same failure mode baked into the operator's math itself, not just its implementation: because each output channel is convolved with only its own single small filter (no reduction across channels), there is very little arithmetic to amortize each loaded byte against — we'll derive this precisely in Section 2.4, and it is a large part of why depthwise-separable architectures (MobileNet-style) trade FLOP count for memory-bound kernels that don't always deliver the latency win their FLOP counts promise on bandwidth-starved edge hardware.

The rest of this chapter builds the tool that turns "probably compute-bound" or "probably memory-bound" into an exact, falsifiable number.

2. Arithmetic Intensity: Putting a Number on "How Much Work Per Byte"

2.1 Definition and a first worked example

Arithmetic intensity (AI) is the ratio of computational work performed to data moved to perform it:

AI = Operations / Bytes Transferred

measured in FLOP/byte (or, for integer kernels, ops/byte). "Bytes transferred" means bytes crossing the boundary between the compute unit and the memory tier being analyzed — most commonly DRAM, since that is usually the scarcest, slowest tier, though Section 4 revisits this choice.

A minimal worked example: suppose a kernel performs 2×10^9 operations while moving 500×10^6 bytes across the memory bus to do so. Then:

AI = 2×10⁹ operations / 500×10⁶ bytes
   = 4 operations per byte

An AI of 4 FLOP/byte is, on most modern accelerators, quite low — as the next section shows, many devices need AI in the tens or hundreds of FLOP/byte before the compute units become the bottleneck. This single number is the whole diagnostic: low AI predicts memory-bound behavior regardless of how large the raw operation count looks, and high AI predicts compute-bound behavior. The rest of this section derives AI precisely for real operators instead of leaving it as an abstract ratio.

2.2 Worked derivation: naive vs. tiled dense matmul

Take a concrete, representative case: C = A × B, all three matrices square, N = 1024, float32 (4 bytes per element). This is exactly the shape underneath a large fully-connected layer.

FLOP count. Each output element C[i][j] is an inner product of length N: N multiplies and N adds, i.e. 2N FLOPs. There are N² output elements:

FLOPs = 2 × N × N × N = 2N³
      = 2 × 1024³
      = 2 × 1,073,741,824
      = 2,147,483,648 FLOPs
      ≈ 2.147 GFLOP

Byte count — the naive case (no reuse). In a pathologically naive implementation, the triple-nested loop re-fetches A[i][k] and B[k][j] from DRAM on every single iteration of the inner product, because nothing about the loop order or cache keeps a previously-loaded value resident by the time it's needed again. Each of the N³ multiply-accumulate steps then costs 2 loads of 4 bytes each; the final store of each C[i][j] happens once, after its accumulation completes:

Bytes_naive = N³ × (4 bytes for A[i][k] + 4 bytes for B[k][j])  +  N² × 4 bytes (store C)
            = 1,073,741,824 × 8  +  1,048,576 × 4
            = 8,589,934,592  +  4,194,304
            = 8,594,128,896 bytes
            ≈ 8.59 GB

AI_naive = 2,147,483,648 FLOPs / 8,594,128,896 bytes
         ≈ 0.250 FLOP/byte

That is a strikingly poor ratio — worse, in fact, than several of the "obviously" memory-bound elementwise kernels below. This is the counterintuitive point flagged in Part 1's framing: a raw O(N³) operation count guarantees nothing about arithmetic intensity. It only pays off if the implementation reuses data.

Byte count — the tiled, near-ideal case. A well-tiled (blocked) GEMM restructures the same three loops so that sub-blocks of A, B, and C small enough to fit in L1 or L2 (or, on an NPU, local SRAM) are loaded once and reused for every MAC that touches them before being evicted. In the theoretical limit — each element of A, B, and C crossing the DRAM boundary exactly once, which a good enough tile size and loop order approaches — total traffic is just the three matrices themselves:

Bytes_ideal = 3 × N² × 4 bytes
            = 3 × 1,048,576 × 4
            = 12,582,912 bytes
            ≈ 12.58 MB

AI_ideal = 2,147,483,648 FLOPs / 12,582,912 bytes
         ≈ 170.67 FLOP/byte

That's a 683× improvement in arithmetic intensity from tiling alone, with the exact same FLOP count. No new arithmetic was invented; the only thing that changed is how many times each byte was allowed to do work before being discarded. Note the general shape of this result: for an N×N×N matmul in fp32 with ideal reuse,

AI_ideal(N) = 2N³ / (3 × 4N²) = N / 6

Arithmetic intensity of a well-tiled matmul grows linearly with matrix dimension — bigger GEMMs get relatively more compute-bound, which is exactly why the large GEMMs inside transformer feed-forward layers and attention projections are the operators NPU vendors optimize hardest for, and why tiny GEMMs (e.g. a batch-1 decode step with a short sequence) are comparatively harder to keep out of the memory-bound regime.

2.3 Worked derivation: elementwise add, revisited

If the memory hierarchy chapter in the companion Computer Architecture series is familiar, this is the same kernel that motivated the memory wall discussion there — worth re-deriving here as the calibration point for "unambiguously memory-bound." Take C[i] = A[i] + B[i] in float32:

Operations per element = 1 (one floating-point add)
Bytes per element:
    load  A[i]  = 4 bytes
    load  B[i]  = 4 bytes
    store C[i]  = 4 bytes
    total       = 12 bytes
 
AI = 1 / 12 ≈ 0.083 FLOP/byte

This ratio is independent of array length N — it's a fixed property of the operation's structure, not something a bigger problem size fixes the way it did for matmul. There is no tiling scheme that improves it, because there is no reuse to exploit: every byte loaded is used exactly once and never touched again. This is the ceiling case for "the algorithm itself has no data-reuse structure to recover," and it is why elementwise ops — activation functions, bias adds, residual connections, normalization scale/shift — are treated in production inference runtimes as things to fuse into their neighbors rather than optimize in isolation. There's no isolated optimization available; the only lever is not paying the DRAM round-trip at all (Section 5).

2.4 Worked derivation: depthwise vs. regular convolution

This pair makes the "operator math itself, not just the implementation" version of low arithmetic intensity concrete. Take a feature map of spatial size 112×112, float32, with a 3×3 kernel and "same" padding (stride 1), and compare a depthwise convolution against a regular (dense) convolution over it.

Depthwise convolution, 32 input channels, 32 output channels (one 3×3 filter per channel, no cross-channel reduction):

FLOPs per output element = 3×3 kernel taps × 2 (multiply + add) = 18
Output elements          = 112 × 112 × 32 = 401,408
 
FLOPs = 401,408 × 18 = 7,225,344 FLOPs ≈ 7.23 MFLOP
 
Bytes (ideal, each tensor touched once):
    input  = 112 × 112 × 32 × 4 bytes = 1,605,632 bytes
    weights= 32 × 3 × 3 × 4 bytes     =     1,152 bytes
    output = 112 × 112 × 32 × 4 bytes = 1,605,632 bytes
    total                              = 3,212,416 bytes ≈ 3.06 MB
 
AI_depthwise = 7,225,344 / 3,212,416 ≈ 2.25 FLOP/byte

Regular convolution, 32 input channels → 64 output channels, same spatial footprint (every output channel is a weighted sum across all input channels):

FLOPs per output element = Cin × kernel taps × 2 = 32 × 9 × 2 = 576
Output elements          = 112 × 112 × 64 = 802,816
 
FLOPs = 802,816 × 576 = 462,422,016 FLOPs ≈ 462.4 MFLOP
 
Bytes (ideal, each tensor touched once):
    input  = 112 × 112 × 32 × 4 bytes =  1,605,632 bytes
    weights= 64 × 32 × 3 × 3 × 4 bytes=     73,728 bytes
    output = 112 × 112 × 64 × 4 bytes =  3,211,264 bytes
    total                              =  4,890,624 bytes ≈ 4.66 MB
 
AI_regular = 462,422,016 / 4,890,624 ≈ 94.56 FLOP/byte

Same spatial resolution, same-order byte traffic (3.06 MB vs. 4.66 MB — less than 1.6× apart), but a 64× difference in FLOP count and a 42× difference in arithmetic intensity. The regular convolution's cross-channel reduction gives every loaded weight and activation far more work to do before it's discarded; the depthwise convolution, by construction, never mixes channels, so each output only ever draws on 9 weight values and a small neighborhood of one channel. This is precisely the mechanism referenced in Section 1.3: depthwise-separable convolutions cut FLOP count aggressively (which is why they show up in mobile-oriented architectures), but the FLOP reduction outpaces the byte reduction, so arithmetic intensity — and therefore achievable hardware utilization — drops sharply alongside it. A FLOP-count comparison alone would call the depthwise layer "28× cheaper"; it does not by itself tell you it will also be far harder to run near peak throughput.

2.5 Summary table

Two things should jump out. First, the same operator (C = A×B) appears twice in this table at wildly different arithmetic intensities — AI is a property of an operator's implementation, not of the operator in the abstract, which is the central warning of this whole section. Second, "memory-bound" and "compute-bound" aren't absolute labels attached to a number in isolation — 2.25 FLOP/byte is memory-bound on a device with a high compute-to-bandwidth ratio and could be closer to compute-bound on a device with less raw arithmetic throughput per byte/s of bandwidth. That relationship — AI compared against a specific piece of hardware — is exactly what the Roofline model formalizes next.

3. The Roofline Model, Derived From First Principles

3.1 Setting up the two roofs

A piece of hardware has exactly two relevant peak numbers for this analysis:

• P_peak — the peak arithmetic throughput of its compute units, in FLOP/s (or a fused metric like TFLOP/s). This is a hard ceiling set by transistor count, clock frequency, and per-cycle throughput per lane.
• B_memory — the peak bandwidth of whichever memory tier is feeding those compute units, in GB/s. This is a hard ceiling set by bus width, memory clock, and channel count.

Both are properties of the silicon, fixed regardless of what kernel is running. A given kernel, running at some measured or predicted arithmetic intensity AI, can never exceed either ceiling — the question the Roofline model answers is which ceiling actually binds for that kernel.

3.2 Deriving the bound

Assume the best case: compute and memory transfer happen fully overlapped (the hardware prefetches ahead and computes on already-arrived data concurrently with fetching the next block — a realistic assumption for well-pipelined hardware, and the assumption every roofline analysis makes explicit). Under full overlap, total execution time is bounded by whichever of the two — compute or transfer — takes longer, not by their sum:

Time_compute = FLOPs / P_peak
Time_memory  = Bytes / B_memory
Time_total  ≥ max(Time_compute, Time_memory)

Achieved performance is FLOPs divided by time, so:

P = FLOPs / Time_total
  ≤ FLOPs / max(Time_compute, Time_memory)
  = min( FLOPs / Time_compute,  FLOPs / Time_memory )
  = min( P_peak,  FLOPs / (Bytes / B_memory) )
  = min( P_peak,  (FLOPs / Bytes) × B_memory )
  = min( P_peak,  AI × B_memory )

which is the Roofline bound:

P ≤ min(P_peak, B_memory × AI)

Every term is dimensionally consistent — a useful sanity check whenever plugging in real numbers: B_memory in bytes/s times AI in FLOP/byte gives FLOP/s, matching P_peak. The inequality (not equality) matters: this is an upper bound on what the hardware can theoretically deliver, assuming perfect overlap, perfect prefetching, and no other stalls (instruction issue limits, launch overhead, insufficient parallelism to hide pipeline latency). Real measured performance is always somewhere at or below this line — the gap between the roofline bound and what a profiler actually measures is itself a diagnostic, addressed in Section 6.

3.3 The ridge point

The bound has two regimes, and the boundary between them is where the two arguments to min() are equal:

B_memory × AI = P_peak
AI* = P_peak / B_memory

AI* is the ridge point — the arithmetic intensity at which a kernel transitions from memory-bound to compute-bound on a given piece of hardware:

• For AI < AI*: B_memory × AI < P_peak, so the sloped line is the binding constraint. The kernel is memory-bound. Achieved performance scales linearly with arithmetic intensity — double the reuse, double the throughput — and no amount of extra compute silicon changes that until AI crosses the ridge.
• For AI > AI*: B_memory × AI > P_peak, so the flat P_peak ceiling binds instead. The kernel is compute-bound. Increasing arithmetic intensity further buys nothing; the only way to go faster is to raise P_peak itself (more parallelism, higher clock, lower-precision arithmetic with native hardware support) or do fewer FLOPs.

Plotted on log-log axes (performance vs. arithmetic intensity), these two regimes become two straight lines meeting at the ridge point:

The chart earns the name "roofline": the sloped bandwidth line and the flat compute line together form a roof shape, and no achievable kernel can plot a point above either segment.

3.4 Plotting real operators on the chart

Ground this with a concrete device. Take a representative edge NPU: P_peak = 4 TFLOP/s (fp16/int8-fused MAC array) and B_memory = 25.6 GB/s (a single LPDDR4X channel, a realistic edge SoC figure). The ridge point is:

AI* = 4×10¹² / 25.6×10⁹ = 156.25 FLOP/byte

Now place the five operators derived in Section 2 on this device's roofline. For each, achieved performance is min(P_peak, B_memory × AI):

Two operators with identical mathematics — the naive and tiled matmul — land at opposite ends of this chart purely because of implementation. Two operators with a 64× difference in FLOP count — the depthwise and regular convolution — both sit left of the ridge, but the regular convolution reaches 60% of peak while the depthwise convolution reaches barely over 1%. Neither fact is visible from FLOP counts alone; both are immediate once arithmetic intensity is plotted against the hardware's actual ridge point.

It's worth checking how sensitive the ridge point is to hardware choice, because the intuition "bigger accelerator, easier to be compute-bound" is not automatically true. Compare three device classes (figures are representative, not tied to a specific SKU, and vary by vendor and generation):

The edge NPU and the datacenter GPU have nearly identical ridge points despite a ~78× gap in raw compute and a ~80× gap in raw bandwidth — because both scaled up together, the ratio barely moved. The mobile SoC GPU's ridge point is markedly lower, meaning a kernel needs less arithmetic intensity to become compute-bound there than on either of the other two. This is the practically useful takeaway: absolute FLOP/s and GB/s numbers on a spec sheet don't tell you where the ridge point sits — their ratio does, and that ratio is what a given kernel has to be compared against.

4. Hierarchical Rooflines: More Than One Memory Roof

Everything above treated "memory" as a single tier with a single bandwidth number, using DRAM because it's usually the scarcest resource. Real hardware has several memory tiers, each with its own bandwidth, and each imposes its own roofline. On-chip SRAM (or L1/L2 cache) is typically an order of magnitude or more faster than DRAM but far smaller in capacity — the same physics tradeoff behind the general memory hierarchy.

Because of this, a single kernel can be simultaneously "SRAM-roof compliant" and "DRAM-roof violating," or the reverse, and a single DRAM-only roofline can be misleading about where the real bottleneck sits. NVIDIA's Nsight Compute profiler addresses this directly by computing hierarchical rooflines — separate ceilings for L1 cache, L2 cache, and device (DRAM) memory bandwidth, plotted together so a kernel's position relative to each tier is visible at once. Intel Advisor's CPU Roofline analysis similarly distinguishes L1, L2, L3, and DRAM bandwidth ceilings rather than collapsing everything into one number.

For NPU and edge-AI kernel design specifically, this matters because a well-designed dataflow (weight-stationary, output-stationary, or a fused-tile schedule) is explicitly trying to keep most traffic inside the fastest, smallest tier — the accelerator's local SRAM scratchpad — and touch DRAM as rarely as possible. Arithmetic intensity computed against DRAM bytes only can look deceptively low for a kernel that is, from the perspective of the on-chip SRAM roof, comfortably compute-bound; conversely, a kernel can look fine against a generous DRAM AI estimate while thrashing a small L1/L2 tier that a naive analysis never accounted for. The general recipe from Section 3 — AI* = P_peak / B_tier — applies at every level; a full hierarchical roofline is simply that computation repeated once per memory tier, each producing its own sloped line and its own ridge point.

5. Why Memory Movement Dominates: The Fusion Example, Quantified

Section 1.3 introduced the unfused Conv → BatchNorm → Activation chain qualitatively. It's worth putting real numbers on exactly how much traffic operator fusion recovers, because the gap is often surprisingly large relative to how cheap the arithmetic itself is.

Take the regular convolution's output tensor from Section 2.4: 112×112×64, float32, 3,211,264 bytes (≈3.06 MB). A naive, unfused pipeline treats each stage as an independent kernel that must round-trip its input and output through DRAM:

Conv        → writes output to DRAM                    :  3.06 MB write
BatchNorm   → reads conv output, writes BN output       :  3.06 MB read + 3.06 MB write = 6.12 MB
Activation  → reads BN output, writes activation output :  3.06 MB read + 3.06 MB write = 6.12 MB
                                                           ─────────────────────────────
                                          Total traffic for these intermediates: 15.30 MB

A fused kernel — one that keeps the conv output resident in registers or local SRAM, applies the batchnorm affine transform and the activation function on that resident tile, and only then writes the final result to DRAM — needs exactly one write of the final tensor for this same stretch of the pipeline:

Fused Conv+BN+Activation → writes final output to DRAM  :  3.06 MB write

That's a 5× reduction in traffic for this segment of the graph, achieved without changing a single FLOP of arithmetic — the batchnorm scale-and-shift and the activation's comparison-and-select are exactly as expensive as before. The entire savings comes from not paying two DRAM round-trips for data that was going to be consumed again within microseconds anyway. This is the concrete, numeric version of the principle this section is named for: move data as little as possible. It is also precisely why graph-level compilers for inference (XLA, TVM, ONNX Runtime's graph optimizer, and NPU-specific compiler stacks built on MLIR) treat operator fusion as one of the highest-leverage optimizations available — it costs nothing in arithmetic and can move a kernel's effective arithmetic intensity by a large multiplicative factor, exactly the axis the Roofline model says matters most for anything sitting left of the ridge point.

6. Measuring Roofline in Practice: Tools Engineers Actually Use

Everything above computed arithmetic intensity analytically, from operator shapes and dtype sizes. That "ideal" AI assumes perfect reuse — the compiler or hardware achieves exactly the minimum byte traffic the math permits. Real kernels rarely hit that exactly, which is why production profilers report a second, distinct number: achieved AI, measured directly from hardware performance counters that count actual bytes crossing a given memory boundary, cache effects and all.

Intel Advisor's Roofline feature runs a Survey analysis followed by a Trip Counts/FLOP analysis on CPU or GPU code, then plots every loop or function in the program as a single dot on a roofline chart — position given by that loop's measured FLOP count and measured memory traffic, not an analytical estimate. Loops sitting well below the roofline they should theoretically reach (a "wasted" gap between the dot and the line) are flagged as latency-bound rather than bandwidth- or compute-bound — a case the pure Roofline inequality from Section 3.2 doesn't capture, since it assumed perfect overlap.

NVIDIA's Nsight Compute computes an equivalent per-kernel roofline for CUDA kernels as part of its "GPU Speed Of Light" section, and — as described in Section 4 — extends it to a hierarchical view across L1, L2, and device memory rather than a single DRAM-only line. Because it derives bytes moved from hardware counters, it captures real effects an analytical calculation misses entirely: cache line granularity, memory coalescing efficiency, bank conflicts, and partial-line waste all show up as achieved AI being lower than the ideal AI computed by hand.

The gap between a hand-derived ideal AI (Section 2's numbers) and a profiler's measured AI is itself a useful diagnostic: a large gap on a kernel that was designed to be compute-bound is a strong signal that the implementation — tiling, layout, or access pattern — is leaving reuse on the table that the algorithm's math actually supports. This is exactly the tiled-vs-naive matmul gap from Section 2.2, expressed as something a profiler would show directly on real hardware rather than something derived on paper.

7. Autoregressive Decode: Arithmetic Intensity at Batch Size One

The single most consequential real-world instance of the memory-bound regime in 2026 is autoregressive LLM decoding, and it's worth deriving explicitly because the result is more extreme, and more structurally unavoidable, than any example above.

During single-token decode (batch size 1, generating one new token per forward pass), each linear projection in the network is not a matrix-matrix multiply — it's a matrix-vector multiply (GEMV): a [1, d_model] activation vector against a [d_model, d_model] (or [d_model, d_ff]) weight matrix. Derive its arithmetic intensity the same way as Section 2.2, keeping the element size S bytes symbolic (S = 4 for fp32, 2 for fp16/bf16, 1 for int8):

y[d_out] = W[d_out, d_in] · x[d_in]
 
FLOPs = 2 × d_out × d_in                       (one multiply + one add per weight)
 
Bytes (weight matrix dominates; x and y are tiny by comparison and can be ignored):
    Bytes ≈ d_out × d_in × S
 
AI = FLOPs / Bytes
   = (2 × d_out × d_in) / (d_out × d_in × S)
   = 2 / S

Every dimension-dependent term cancels. Arithmetic intensity for batch-1 GEMV depends only on the element size, never on model width or depth. For fp16 weights, AI = 2/2 = 1 FLOP/byte; for int8, AI = 2/1 = 2 FLOP/byte. Compare either number against the edge NPU's ridge point of 156.25 FLOP/byte from Section 3.4, or the datacenter GPU's ≈153 FLOP/byte: both are roughly two orders of magnitude below the ridge, on essentially any modern accelerator. This is not a tuning problem an engineer can fix by writing a better kernel — the weight matrix has to be streamed from memory exactly once per token, in full, no matter how the multiply is scheduled, because with only one activation vector there is nothing to reuse a loaded weight against. This is the mathematical root of the widely-observed fact that single-stream LLM decode is bandwidth-bound, not compute-bound, regardless of how large or how well-optimized the model's matrix multiplies are.

The fix follows directly from the roofline logic in Section 3.3: raise AI by amortizing each streamed weight over more than one token. Batching multiple sequences' decode steps together turns the GEMV back into a genuine GEMM — a [B, d_model] activation block against the same [d_model, d_model] weight — and the derivation above simply picks up a factor of B:

AI_batched ≈ (2 × B × d_out × d_in) / (d_out × d_in × S) = 2B / S

Arithmetic intensity scales linearly with batch size, exactly like the tiled-matmul result in Section 2.2 scaled linearly with N. This is the first-principles reason continuous batching and larger decode batch sizes are the primary throughput lever in LLM inference serving systems: it is not a scheduling trick, it is a direct, quantifiable rightward move along the x-axis of the roofline chart, converting a kernel stuck deep in the memory-bound region into one approaching or crossing the ridge point. The tradeoff — larger batches raise throughput but also raise per-token latency and require holding more concurrent KV cache in memory — is the central engineering tension in production LLM serving, and it is a direct, visible consequence of exactly the inequality derived in Section 3.2, not a separate phenomenon requiring its own theory.

8. The Actionable Conclusion

The entire value of this chapter compresses into a decision procedure:

1. Derive (or measure) the kernel's arithmetic intensity: AI = Operations / Bytes.
2. Derive the target hardware's ridge point: AI* = P_peak / B_memory.
3. Compare:
       AI <  AI*  →  memory-bound.  Optimize data movement:
                        tiling, operator fusion, better tensor layout,
                        quantization (fewer bytes per value), exploiting
                        a faster memory tier (Section 4), reducing
                        redundant DRAM round-trips (Section 5).
                      Adding compute throughput will not help.
       AI ≥  AI*  →  compute-bound.  Optimize arithmetic:
                        more parallelism, higher clock, lower-precision
                        MACs with native hardware support, algorithmic
                        FLOP reduction (e.g. Winograd, structured sparsity).
                      Adding memory bandwidth will not help.

One subtlety worth flagging for edge-AI and NPU work specifically: quantization is unusual in that it moves a kernel along both axes at once. Casting activations and weights from fp32 to int8 cuts bytes moved by 4× — a direct rightward shift in arithmetic intensity — while, on hardware with native int8 MAC support, it can simultaneously raise P_peak itself (more int8 MACs fit in the same silicon and power budget than fp32 MACs). That's a rare case where the fix helps on both sides of the inequality in Section 3.2 at once, which is a large part of why aggressive quantization is one of the first levers pulled in edge inference optimization rather than a last resort.

The broader point, stated once more plainly: arithmetic intensity is not a property to compute after profiling reveals a problem — it's a number you can and should derive from an operator's shape and dtype before writing a single line of kernel code, exactly as done in Section 2. It tells you, in advance, which class of optimization is worth your time and which is a waste of it.

Further Reading

• Williams, S., Waterman, A., & Patterson, D., "Roofline: An Insightful Visual Performance Model for Multicore Architectures", Communications of the ACM, Vol. 52, No. 4 (2009) — the canonical paper defining the model, from UC Berkeley's Parallel Computing Laboratory. A freely accessible version is hosted via Berkeley EECS.
• NVIDIA, "Nsight Compute — Roofline Charts" — documentation for the GPU Speed Of Light and hierarchical roofline sections used to profile real CUDA kernels.
• Intel, "CPU Roofline Report Overview — Intel Advisor" — documentation for Intel's per-loop roofline profiling workflow (Survey + Trip Counts/FLOP analysis).
• NERSC, "Roofline Performance Model" — a practical, tool-agnostic reference for applying the roofline model to real HPC and scientific kernels.
• Yeh et al., "Hierarchical Roofline Performance Analysis for Deep Learning Applications" (arXiv, 2020) — extends the roofline framework specifically to deep learning workloads across multiple memory tiers.
• Ray, J., "Arithmetic Intensity: Understand Op Limits — Memory or Compute", Better ML (Medium) — an accessible worked-example treatment of arithmetic intensity for ML operators.
• Lienhart, P., "LLM Inference Series: 5. Dissecting Model Performance" (Medium) — applies the roofline framework directly to transformer decode, including the batch-size-dependent arithmetic intensity derived in Section 7 above.

This closes Lesson 1. With performance fundamentals and the roofline model in hand, Lesson 2 turns to what an inference compiler actually does with this information: tensor layouts, tiling strategies, and graph-level optimization.