Parallelism, the Modern CPU, and the Big-Picture Mental Model

This is Lesson 1, Part 10 — the closing chapter.

Parts 1 through 9 built up number representation, gates, the ALU, sequential logic, registers and the PC, the control unit, fetch-decode-execute, pipelining and hazards, the memory hierarchy, and the ISA/compiler boundary, one piece at a time.

This part's job is to weave all of it into a single mental model, go deep on parallelism — the one topic we've only touched in passing — and hand you off to Lesson 2, where we actually build a CPU.

1. Where We've Been, in One Pass

Before going forward, compress everything backward.

A number is a pattern of bits.

A gate turns patterns of bits into other patterns of bits, with no memory of the past.

An ALU is a pile of gates arranged to compute arithmetic and logic.

A flip-flop is a circuit that remembers one bit across time, using feedback.

A register is a bank of flip-flops that remembers one word.

A clock is the heartbeat that says now it is safe to update memory.

A control unit is a circuit that reads an instruction's bits and decides which other circuits get to act this cycle.

Fetch-decode-execute is the loop that repeats every cycle.

A pipeline overlaps that loop so multiple instructions are mid-flight at once, and hazards are the ways overlapping breaks correctness.

A cache is a small fast memory that exploits the fact that programs re-touch the same data and the same neighborhoods of data.

An ISA is the contract between software and hardware.

A compiler is the machine that translates human intent into that contract.

Every one of those ideas is still just "bits, gates, and a clock."

Nothing new has been invented since Part 1 — the rest of the series is elaboration, not addition.

Keep that in your head as we go through parallelism, because it's tempting to treat SIMD units and multicore chips as some separate category of magic.

They aren't.

They're the same gates and the same clock, replicated and coordinated.

2. The Four Kinds of Parallelism

"Parallelism" is one word covering four genuinely different mechanisms.

Conflating them is the single most common source of confusion when people first read architecture papers, so it's worth being pedantic here.

2.1 Instruction-Level Parallelism — ILP

ILP means multiple instructions from the same instruction stream execute simultaneously.

This happens because a single core has multiple execution units and can issue more than one instruction per cycle (superscalar), and can execute instructions out of program order once their operands are ready (out-of-order execution, which Part 7's discussion of hazards touched on, and which a future phase will cover properly with Tomasulo's algorithm).

I1 (ADD)  ─────────►
I2 (MUL)  ─────────►    all three in flight in the same core,
I3 (LOAD) ─────────►    on the same cycle, from the same program

ILP is invisible to the programmer.

You write sequential code; the hardware finds independence and exploits it.

That invisibility is exactly what makes it hard to build — the hardware has to discover parallelism the compiler didn't have to prove.

2.2 Data-Level Parallelism — DLP

DLP means the same operation applies to many data elements at once.

This is what SIMD — "Single Instruction, Multiple Data," one of the four categories in Flynn's taxonomy — and vector units implement:

A0  A1  A2  A3
 +   +   +   +
B0  B1  B2  B3
 =   =   =   =
C0  C1  C2  C3      one instruction, four additions

Unlike ILP, DLP is not invisible.

The programmer, or the compiler's auto-vectorizer, has to explicitly arrange data into contiguous, uniformly-typed groups and issue a vector instruction.

We'll spend the next section going deep on exactly how this works, because it's the parallelism form most directly under your control as a systems programmer.

2.3 Thread-Level Parallelism — TLP

TLP means independent streams of instructions — threads — execute on separate cores simultaneously.

Core 0 → Thread A
Core 1 → Thread B
Core 2 → Thread C
Core 3 → Thread D

Each core has its own fetch/decode/execute machinery, its own program counter, its own pipeline.

TLP is where "more cores" lives, and it's also where Amdahl's Law lives — which is the whole reason "just add more cores" doesn't scale the way people intuitively expect.

More on that below.

2.4 Task-Level Parallelism

Task-level parallelism is the coarsest grain: independent tasks run concurrently.

Each task might itself be single-threaded, multi-threaded, or offloaded to an accelerator — a web server handling separate requests, a build system compiling separate translation units, a video pipeline running decode on the CPU while encode runs on a dedicated ASIC.

It overlaps conceptually with TLP but operates at the level of whole jobs rather than instruction streams sharing one address space.

Four different problems, four different mechanisms. ILP is the hardware finding independence within one instruction stream. DLP is one instruction acting on many data elements. TLP is many instruction streams on many cores. Task-level parallelism is many independent jobs. Knowing which one a given optimization or bottleneck belongs to is half of solving it.

3. SIMD in Depth: Doing the Arithmetic on "Vector Width"

Let's actually build the numbers, because "SIMD is faster" is a claim you should never accept without doing the arithmetic yourself.

3.1 The Setup

Take the simplest possible DLP kernel: element-wise addition of two float arrays.

void add_arrays_scalar(const float* a, const float* b, float* c, int n) {
    for (int i = 0; i < n; ++i) {
        c[i] = a[i] + b[i];
    }
}

On a scalar core, each loop iteration is roughly: load a[i], load b[i], add, store c[i], increment i, compare against n, branch back.

Call that six instructions per element.

The exact count depends on the ISA and how aggressively the compiler unrolls, but six is a reasonable first-principles estimate for a naive translation.

For n = 1,000,000 elements, that's roughly six million instructions executed.

Now the NEON version, targeting a 128-bit vector register that holds four 32-bit floats at once.

NEON is the vector ISA on Cortex-A and Apple Silicon cores, and functionally the same idea as x86's SSE/AVX — just a different register width.

#include <arm_neon.h>
 
void add_arrays_neon(const float* a, const float* b, float* c, int n) {
    int i = 0;
    for (; i + 4 <= n; i += 4) {
        float32x4_t va = vld1q_f32(&a[i]);   // load 4 floats from a
        float32x4_t vb = vld1q_f32(&b[i]);   // load 4 floats from b
        float32x4_t vc = vaddq_f32(va, vb);  // add all 4 lanes in one instruction
        vst1q_f32(&c[i], vc);                // store 4 results
    }
    for (; i < n; ++i) {                     // remainder loop for n % 4 != 0
        c[i] = a[i] + b[i];
    }
}

3.2 The Actual Arithmetic

The vector loop body is still one load, one load, one add, one store, one increment/compare/branch — six instructions.

But now each pass handles four elements instead of one.

For n = 1,000,000:

scalar:  1,000,000 iterations × 6 instructions  = 6,000,000 instructions
NEON:      250,000 iterations × 6 instructions  = 1,500,000 instructions
 
instruction-count reduction = 6,000,000 / 1,500,000 = 4.0×

If every instruction still costs roughly one cycle of issue/execute throughput — a reasonable approximation when the ALU/load-store pipeline isn't the bottleneck — the vectorized loop needs about a quarter of the cycles.

That's Speedup ≈ vector_width, here 4×, for a purely compute-bound kernel where instruction issue is the limiting resource.

3.3 Where the 4× Falls Apart

That number is the ceiling, not the guarantee, and it's worth being explicit about why real measurements come in lower.

This kernel is memory-bound, not compute-bound.

Each element does one add but touches 12 bytes of memory — two 4-byte loads, one 4-byte store.

Real cores can usually issue vector loads/stores faster than DRAM, or even L2, can sustain the resulting bandwidth.

If the memory system is the bottleneck, quadrupling the arithmetic rate doesn't quadruple the program's speed — the loads and stores were already the limiter, and DLP didn't make DRAM four times faster.

This is the same "constraint forces a mechanism" story as caching, just showing up as a ceiling on a different optimization.

The remainder loop matters too.

If n isn't a multiple of 4, the last n % 4 elements run scalar.

For large n this is negligible; for small n it can dominate.

Misaligned or non-contiguous data can force slower load/store variants or block vectorization outright — this is why array-of-structs layouts are notoriously bad for SIMD compared to struct-of-arrays.

Not every operation vectorizes cleanly.

Data-dependent branches inside the loop, pointer aliasing the compiler can't rule out, and reductions with dependencies between iterations all reduce or block the achievable speedup.

The lesson isn't "SIMD gives you N×." It's "SIMD gives you up to N× on the fraction of the program that is (a) vectorizable and (b) not bottlenecked on something else." Both qualifications matter, and the second one is exactly Amdahl's Law, which is the next topic — and it applies just as much to a single vector instruction's fraction-of-runtime as it does to spreading work across cores.

3.4 Vector Width Is Just Another Design Knob

The width itself is a hardware decision under the same die-area and power constraints as everything else in this series.

NEON is 128 bits — 4×float32 or 2×float64.

AVX2 is 256 bits — 8×float32.

AVX-512 is 512 bits — 16×float32.

ARM's SVE is variable-length, letting the same binary run on implementations with different physical vector widths.

Wider vectors mean more parallelism per instruction, but also more silicon devoted to the vector register file and execution units, more power drawn when those lanes are active — wide-vector instructions are a well-known cause of frequency throttling on real x86 chips — and diminishing returns once you're memory-bound.

It's the same tradeoff shape as everything else we've studied: performance in one dimension bought with area and power in another.

A quick comparison across real ISAs makes the scaling concrete:

Each step down that table is roughly a doubling of theoretical throughput on a compute-bound kernel, and roughly a doubling of the silicon and power budget devoted to the vector unit.

SVE's variable length is itself a first-principles response to a real problem: fixed-width ISAs like AVX force every future chip generation that wants a wider vector unit to add an entirely new instruction set extension — MMX, then SSE, then AVX, then AVX-512, each with its own encoding — while software has to be recompiled or multi-versioned to use the new one. A length-agnostic ISA lets the same compiled binary run correctly, and get proportionally faster, on hardware with a wider physical vector unit, without a recompile.

4. Multicore Is Not Automatically Faster: Amdahl's Law

Here is a fact that surprises people who haven't derived it themselves: doubling the number of cores essentially never doubles the speed of a real program.

The reason is simple once you see it, and it's one of the most important results in all of computer architecture — Gene Amdahl's 1967 observation, still taught in essentially every architecture course, including MIT's 6.004/6.823 sequence.

4.1 The Derivation

Split any program's execution time into two parts.

A fraction P can be parallelized across N processors.

The remaining fraction (1 - P) is inherently sequential — I/O, setup, a critical section, a dependency chain that cannot be split no matter how much hardware you throw at it.

total time (1 core)      = (1 - P) + P
total time (N cores)     = (1 - P) + P / N        (the parallel part splits N ways;
                                                     the sequential part does not shrink)
 
Speedup = time(1 core) / time(N cores)
 
Speedup = 1 / ((1 - P) + P / N)

That's Amdahl's Law.

Read the denominator carefully: as N grows without bound, P / N shrinks toward zero, and the whole expression converges to 1 / (1 - P) — a hard ceiling set entirely by the sequential fraction, no matter how many cores you add.

4.2 Worked Example

Take a program where 95% of the runtime is parallelizable (P = 0.95) — a generous, optimistic fraction for real software.

Notice the shape.

Doubling from 1 to 2 cores buys 0.90× of extra speedup.

Doubling from 32 to 64 buys only 2.87× of extra speedup.

No matter how many cores you throw at this program, it will never exceed 20× — even with an infinite number of cores.

That 20× ceiling comes entirely from the 5% that can't be parallelized: 1 / (1 - 0.95) = 20.

Now contrast with a program that's only 50% parallelizable:

Speedup(N=2)   = 1 / (0.5 + 0.5/2)   = 1 / 0.75  = 1.33×
Speedup(N=8)   = 1 / (0.5 + 0.5/8)   = 1 / 0.5625 = 1.78×
Speedup(N=∞)   = 1 / 0.5             = 2.00×

A 64-core machine running this program is barely faster than a 2-core machine running it.

The ceiling is 2× regardless of core count, because half the work simply cannot be split.

The takeaway isn't "parallelism doesn't work." It's "the payoff from N cores is bounded by the part of the program you didn't parallelize, and that bound shows up fast." This is precisely why real engineering effort on multicore software goes into shrinking the sequential fraction — reducing lock contention, eliminating serialized I/O, breaking dependency chains — rather than blindly adding threads.

4.3 Why This Connects Back to SIMD and Caches

Amdahl's Law isn't specific to multicore — it's a statement about any optimization that only speeds up part of a system.

Section 3's SIMD kernel obeys exactly the same law.

If the vectorizable loop is 80% of a program's runtime and SIMD gives that loop a 4× speedup, the whole program's speedup is 1 / (0.2 + 0.8/4) = 1 / 0.4 = 2.5×, not 4×.

A bigger cache obeys the same law.

If only 30% of runtime is spent stalled on cache misses, even a mythical zero-latency cache can only ever buy you 1 / 0.7 ≈ 1.43×.

Every performance mechanism in this series — cache, prediction, out-of-order execution, wider pipelines, more cores, wider vectors — ultimately answers to Amdahl's Law, because every one of them speeds up some fraction of execution while leaving the rest untouched.

5. A Modern CPU Is Really Many Machines Working Together

With ILP, DLP, and TLP named and the Amdahl ceiling understood, look again at what a real high-performance core actually is.

It is not ALU + registers.

It's closer to several cooperating machines, each solving one of the constraint-driven problems this series has walked through:

The front end's job is to keep the execution units fed despite branches (hence prediction) and instruction-cache misses.

The execution stage's job is to run as many independent operations per cycle as it can find — hence superscalar issue and SIMD lanes sitting alongside plain integer ALUs.

The memory subsystem's job is to hide DRAM latency behind a hierarchy of progressively larger, slower caches.

Retirement's job is to make out-of-order, speculative execution look, from software's point of view, exactly like the strictly sequential fetch-decode-execute loop from Part 6 — nothing is allowed to become externally visible out of program order, even though internally almost nothing happens in program order.

Zoom out one more level, and a chip is several of these cores wired together:

and one more level past that:

Memory Controller
       │
       ▼
     DRAM
       │
       ▼
      I/O
       │
       ▼
SSD / GPU / NIC / other peripherals

Every box in these three diagrams is something Lesson 1 has already given you a first-principles reason for.

There is no box here that exists "because that's how CPUs are built."

Each one exists because a specific constraint — memory is slow, branches are unpredictable in advance, one ALU isn't enough throughput, DRAM bandwidth is finite — forced a specific mechanism into existence.

That reframe is worth stating as its own principle, because it's the single highest-leverage habit you can take out of this series.

6. The Reframe: Constraints Force Mechanisms

Don't think of computer architecture as a list of components to memorize.

Think of it as a chain of problem → forced solution, where each solution introduces new state, new hardware, and usually a new problem one level down.

Every mechanism in this table also introduces a new problem.

Caches introduce coherence problems across cores.

Branch prediction introduces misprediction penalties.

Out-of-order execution introduces the need for precise exceptions and retirement.

Register renaming introduces the need for a larger physical register file.

Architecture, viewed this way, is an unbroken chain of "solve this, and here's the next thing you now have to solve."

7. The Seven Fundamental Architecture Questions

For any mechanism in that chain, ask these seven questions.

This framework generalizes far beyond this series — it's the same set of questions you'd bring to reading an actual microarchitecture manual or a conference paper.

1. What problem does it solve?
2. What state does it maintain?
3. What hardware implements it?
4. What is its latency?
5. What is its throughput?
6. What happens on failure?
7. What tradeoff does it introduce?

We used a light version of this framework implicitly throughout the series for caches.

Let's now apply it fully, end to end, to a mechanism this series has mentioned constantly but never actually built: the branch predictor.

7.1 Branch Prediction, Fully Worked

1. What problem does it solve?

In a pipelined core (Part 7), fetch happens several stages ahead of execute.

A conditional branch's outcome — taken or not-taken, and its target address — usually isn't known until it's evaluated, which can be several stages after it was fetched.

If fetch simply waits, every branch stalls the pipeline for the number of stages between fetch and resolution.

On real pipelines that's easily 15-20 cycles of dead time per branch, and branches occur roughly every 5-6 instructions in typical code.

Left unaddressed, this alone would cut effective throughput by more than half.

2. What state does it maintain?

The simplest useful predictor is a table of small counters, one per sampled branch, called a Branch History Table (BHT) or Pattern History Table (PHT) depending on the design.

Each entry is a 2-bit saturating counter with four states:

Each outcome nudges the counter one step toward the corresponding extreme.

It increments on taken, saturating at 11, and decrements on not-taken, saturating at 00.

The predicted direction is simply read off the top bit.

Alongside the counter table, real cores also keep a Branch Target Buffer (BTB) that caches the target address of recently seen branches, so the front end knows not just "taken or not" but where to fetch from next, without waiting to decode the branch instruction itself.

The reason it's 2 bits and not 1 is a concrete, memorable example.

Consider a loop that runs, say, 9 times taken and exits once not-taken, over and over — a very common pattern, think of any fixed-trip-count inner loop called repeatedly.

With a 1-bit predictor, the single not-taken exit flips the bit to "predict not-taken."

The next time the loop is entered, the first iteration is again taken — but the predictor now says not-taken, so it mispredicts again, right at loop re-entry, before flipping back.

That's two mispredictions per loop invocation.

With a 2-bit counter, the lone not-taken exit only nudges the counter from "strongly taken" to "weakly taken" — it does not flip the predicted direction.

The next loop entry is still predicted taken, correctly.

The 2-bit counter adds hysteresis: it takes two consecutive contrary outcomes to actually change the prediction, not one, which is exactly the behavior this common loop shape needs.

3. What hardware implements it?

A small SRAM array — often a few hundred to a few thousand entries, each 2 bits wide — indexed by low-order bits of the program counter in the simplest scheme, read out combinationally alongside instruction fetch.

There's no tag comparison in the simplest design, unlike a cache, which means different branches can alias to the same counter — a deliberate area/accuracy tradeoff, addressed below.

The BTB, by contrast, typically is tag-checked, structured much like a small cache, since a wrong target is far more costly to act on than a slightly wrong direction bit.

4. What is its latency?

The prediction must be available in the same cycle as fetch, or the very stall this mechanism exists to eliminate reappears.

That single constraint is why the table is small, tagless, and directly indexed rather than associative — anything requiring multi-cycle lookup defeats the purpose.

5. What is its throughput?

Modern superscalar front ends fetch multiple instructions — and therefore potentially multiple branches — per cycle, 4 to 8 wide on high-end cores.

The predictor has to sustain a prediction, and a BTB lookup, per fetch bundle per cycle.

In practice that means the structure is banked or pipelined so it doesn't itself become the bottleneck it was built to remove.

6. What happens on failure, i.e. a misprediction?

Every instruction fetched, decoded, and speculatively executed past the mispredicted branch is wrong and must be discarded.

 fetch → decode → issue → execute → ... → retire
   │
   ▼ (branch resolves here, several stages later)
   mispredict detected
   │
   ▼
 flush every instruction fetched after the branch
   │
   ▼
 refetch from the correct target

The cost is the full distance between fetch and branch resolution — the misprediction penalty — often 15-20 cycles on deep, high-frequency pipelines.

This is precisely why prediction accuracy matters so much.

Modern predictors routinely exceed 95% accuracy, and even that leaves a real cost, because the 5% failures are each expensive.

7. What tradeoff does it introduce?

A bigger table reduces destructive aliasing between unrelated branches — fewer distinct branches sharing one counter — but costs more SRAM area and more read-latency pressure, the same area/latency tension that showed up for caches.

Using global branch history — recent outcomes of other branches, not just this one — to index the table can capture correlations ordinary per-branch counters miss, for example predicting one branch based on how a preceding, related branch just resolved.

That costs more state and more complex indexing.

And the 2-bit-vs-1-bit choice itself is a tradeoff: 2 bits doubles the storage per entry but eliminates exactly the loop-exit misprediction pattern shown above.

Every one of these is the same shape of tradeoff you've now seen for cache size, pipeline depth, and vector width: more accuracy or more capability, paid for in area, power, or latency.

Running any mechanism through these seven questions turns a black box into an engineering artifact with legible constraints. Try it next on the memory hierarchy's replacement policy, or on register renaming, using only what Parts 1-9 already gave you.

8. The Big Picture: One Mental Hierarchy

Zoom all the way out.

This is the stack Lesson 1 has been assembling, piece by piece, since Part 1:

Every layer in this diagram is a translation.

Application intent becomes C++.

C++ becomes IR.

IR becomes assembly.

Assembly becomes ISA-encoded bits.

Those bits get interpreted by a control unit built from gates.

The gates are switches made of doped silicon obeying electromagnetism.

Nothing in this stack is arbitrary, and — this is the point of Lesson 1 — nothing in it should feel like a black box to you anymore.

ADD R3, R1, R2 at the top of this stack is voltages settling on wires at the bottom, and you now have every layer connecting the two.

9. Closing Exercise: One Instruction, All the Way Down

Before Lesson 2, do this once, on paper, without looking anything up.

Take:

ADD R3, R1, R2

and trace it through every layer:

PC → Instruction Memory → Instruction bits → Decoder →
Register addresses → Register File → Two operands →
ALU → Result → Register File → R3

Now ask, at every single arrow: what physical hardware exists here?

Register File → ALU isn't a metaphorical connection.

If registers are 32 bits wide, there are 32 physical signal paths per operand carrying real voltages:

The ALU is built from the full-adder and gate logic Parts 2-3 derived.

The result rides back on 32 more physical wires.

The destination register is a bank of flip-flops — actual storage elements built from cross-coupled gates, from Part 4.

The clock edge is the specific moment those flip-flops are permitted to latch the new value.

Trace this all the way through and you've personally rebuilt the bridge between "software instruction" and "physical computation" that this entire lesson exists to establish.

Once "software instruction" and "physical computation" stop feeling like two different worlds, the rest of computer architecture stops being a list of facts to memorize and becomes a set of consequences you can derive. That shift is the actual goal of Lesson 1 — everything else was in service of it.

Further Reading

• Hennessy, J. L. and Patterson, D. A., Computer Architecture: A Quantitative Approach, 6th ed., Morgan Kaufmann / Elsevier — the standard graduate reference for everything in this post, especially Amdahl's Law, superscalar/out-of-order execution, and branch prediction. Elsevier product page

• Patterson, D. A. and Hennessy, J. L., Computer Organization and Design, 6th ed. (MIPS/ARM/RISC-V editions), Morgan Kaufmann / Elsevier — the undergraduate companion, with a gentler treatment of the fetch/decode/execute-to-pipelining path this series has followed. Elsevier product page

• MIT OpenCourseWare, 6.004 Computation Structures — Lecture 21, "Parallel Processing," covers ILP/DLP/TLP and Amdahl's Law with the same annotated-slide style as the rest of the course. ocw.mit.edu

• MIT OpenCourseWare, 6.823 Computer System Architecture — a full lecture note set covering superscalar issue, out-of-order execution, and branch prediction in more depth than this post attempts. ocw.mit.edu

• GeeksforGeeks, "Amdahl's Law and its Proof" — a compact derivation and worked examples matching the one in Section 4. geeksforgeeks.org

• GeeksforGeeks, "Solution of Control Dependency" — covers branch prediction and the control-hazard problem it solves, from the pipelining side rather than the predictor-internals side used here. geeksforgeeks.org

What's Next: Building a Real CPU

Lesson 1 was deliberately conceptual — every diagram in this series so far has been a block diagram, not a schematic.

That changes immediately.

Lesson 2, "Designing a Single-Cycle CPU," starts from a blank sheet and builds an actual, working CPU design.

A tiny ISA, designed from scratch, with a handful of instructions — something in the shape of ADD, SUB, AND, OR, LW, SW, BEQ — small enough to fully specify in one sitting, large enough to be a real, Turing-complete instruction set.

The datapath, built bottom-up exactly the way Part 2 built the ALU: half-adder to full-adder to a 32-bit ALU, then MUXes and a register file and a program counter, wired together into the physical paths data actually travels.

The control unit, derived — not guessed — from the ISA's instruction encoding: for each opcode, which control signals must assert, on which cycle, to make the datapath do the right thing.

A full cycle-by-cycle trace of a real program running on that datapath, in the same spirit as this post's closing exercise, but now with a complete, buildable circuit standing behind every arrow instead of a conceptual one.

That is where the roadmap sketched throughout this series continues.

Phase 2, CPU fundamentals, is Lesson 2 in full.

Phases 3 through 8 remain the long-term arc this series is building toward: multi-cycle and pipelined datapaths with real hazard/forwarding logic, the memory hierarchy built down to SRAM cells and TLB structures, superscalar and out-of-order execution with register renaming and reorder buffers, SIMD and multicore taken to the RTL level with cache coherence protocols, GPU/NPU accelerator architecture, and advanced topics like NUMA and interconnects.

Lesson 1 gave you the map.

Lesson 2 is where you start laying track.