Skip to content

// Orchestration · ~2 min read

MapReduce

Reach for it when: you split a large set of documents into chunks, run the same job on each, then combine the results.

After this lesson you can: Run the same job on every chunk of a large input, then combine the results without overflowing one prompt.

// 60-second mental model

How to hold it in your head

Chop the corpus into chunks. Run the same map prompt on every chunk. Reduce the map outputs into one answer. Same job, many shards.

Agent MapReduce splits the input into identical map tasks, runs them (often in parallel), then combines the map outputs with a reduce prompt or plain code.

How this differs

Parallel Fan-Out runs different jobs on the same question. MapReduce runs the same job on different chunks of the input.

// Mini architecture

Partition → identical maps → reduce

A corpus goes to a Partition step that splits it into chunks 1 to n. The same map prompt runs on every chunk. A Reduce step combines the map results into the final summary.

Maps are identical by design. If each shard needs a different specialist, you wanted fan-out with typed branches — not MapReduce.

// Mini-project

Meeting-notes MapReduce (stub)

Chunker + map stubs + reduce stub

Goal: Summarize a long canned transcript by mapping identical extract jobs per chunk, then reducing to decisions and owners.

  1. Split a long stub transcript into fixed-size chunks.
  2. Map: same prompt extracts `{ decisions[], owners[] }` per chunk.
  3. Collect map JSON arrays (no free-text merge yet).
  4. Reduce: dedupe decisions, union owners, emit one brief.
  5. Assert map prompts are byte-identical across chunks.
  6. Log chunk count and reduce input size.

Stub maps may return canned extracts. Homogeneous jobs are the point.

// Common failure

What goes wrong

Symptom

Snowballing reduce — stuffing every map output into one prompt until it overflows.

Fix

Map to structured fields and reduce over those (or reduce in rounds: small groups first, then their results). Cap the size of each reduce call.

// Self-reflection

Sit with this

Where are you summarizing a haystack with one prompt instead of map then reduce?

Sources

Session only. Nothing is saved.