// Orchestration · ~2 min read
MapReduce
Reach for it when: you split a large set of documents into chunks, run the same job on each, then combine the results.
After this lesson you can: Run the same job on every chunk of a large input, then combine the results without overflowing one prompt.
// 60-second mental model
How to hold it in your head
Chop the corpus into chunks. Run the same map prompt on every chunk. Reduce the map outputs into one answer. Same job, many shards.
Agent MapReduce splits the input into identical map tasks, runs them (often in parallel), then combines the map outputs with a reduce prompt or plain code.
How this differs
Parallel Fan-Out runs different jobs on the same question. MapReduce runs the same job on different chunks of the input.
// Mini architecture
Partition → identical maps → reduce
A corpus goes to a Partition step that splits it into chunks 1 to n. The same map prompt runs on every chunk. A Reduce step combines the map results into the final summary.
Maps are identical by design. If each shard needs a different specialist, you wanted fan-out with typed branches — not MapReduce.
// Mini-project
Meeting-notes MapReduce (stub)
Chunker + map stubs + reduce stub
Goal: Summarize a long canned transcript by mapping identical extract jobs per chunk, then reducing to decisions and owners.
- Split a long stub transcript into fixed-size chunks.
- Map: same prompt extracts `{ decisions[], owners[] }` per chunk.
- Collect map JSON arrays (no free-text merge yet).
- Reduce: dedupe decisions, union owners, emit one brief.
- Assert map prompts are byte-identical across chunks.
- Log chunk count and reduce input size.
Stub maps may return canned extracts. Homogeneous jobs are the point.
// Common failure
What goes wrong
Symptom
Snowballing reduce — stuffing every map output into one prompt until it overflows.
Fix
Map to structured fields and reduce over those (or reduce in rounds: small groups first, then their results). Cap the size of each reduce call.
// Self-reflection
Sit with this
Where are you summarizing a haystack with one prompt instead of map then reduce?
Sources
Session only. Nothing is saved.