Don't let the 15% fool you — this domain hides inside every other scenario as the "…and how do you make it production-ready?" follow-up. Context windows, caching, degradation, retries: the unglamorous stuff that decides whether people trust your agent. If Domain 1 is where you design the system, this is where you become the responsible adult who runs it.
📖 New here? The two invisible forces: the window and entropy
Force one — the window. Everything the model "knows" in a call must fit in its context window, and everything competes for it: your instructions, the conversation so far, tool results, documents, and even the space reserved for the answer. Long chats "forget" their beginnings not from amnesia but from economics — something had to go. Nothing falls out by itself, though: in the raw Messages API earlier turns stay until the harness or product actively compacts or drops them; skip that and the request simply exceeds the window and errors (Claude Code auto-compacts as a product behavior). Context management is deciding, deliberately, what earns those tokens: summarize the old, retrieve on demand, isolate heavy work in subagents, and cache what never changes (the single biggest cost lever — repeat calls reuse the unchanged prefix at ~10% price).
Force two — entropy. In the demo, nothing fails. In production, everything does, on schedule: rate limits at your traffic peak, a truncated response the moment JSON matters, a dependency down during the demo to your VP. Reliability here isn't heroics — it's a checklist of boring guarantees: check why generation stopped before trusting it, back off and retry transient errors, make side effects idempotent so retries are safe, degrade gracefully to honest failure, and instrument everything so the 2 AM debug is a query, not an archaeology dig.
So what for the exam: these questions hide inside every scenario as "…and how do you make it production-ready?" The answers are always the same five words deep down: budget, cache, retry, idempotent, instrument.
Context management
🧭 Your seat at the table: this domain is you as the responsible adult of the system — deciding what earns space in the context budget, what happens when things fail (they will), and how you'll know what happened at 2 AM. None of it is glamorous; all of it is what separates a demo from a system people trust.
🧭 Decode every “remembers / forgets” before answering: the model owns the physics — the window's capacity, and attention effects like lost in the middle and recency bias (those genuinely ARE model behavior). Everything else is librarianship, and it is never the model's: every “the agent remembers X” = your harness re-sent X in messages; every “it forgot” = your code dropped/compacted it, or the window overflowed by your arithmetic; prompt caching and server-side compaction = the API acting on your array, under flags you set; the memory tool = the model writes a request for a file operation and deterministic code executes it — the tool-use handshake, yet again. One test for any exam sentence: if the verb is about capacity or attention, the model; if it is about contents, the harness.
Who runs compaction? — the three-tier answer
The question every Claude Code user asks on their way to production: "I run /compact in the terminal — do I do that in my agent too, or is it automatic?" Depends entirely on which tier your system is built on:
You built on
Context management is…
Your job
Claude Code (interactive or headless)
automatic — the product compacts as the window fills; /compact is the manual override
mostly nothing; keep CLAUDE.md lean
Agent SDK
inherited — the same engine's context handling, configurable
tune it; trim your own tool results
Raw Claude API
yours, entirely — the API is stateless; it never summarizes, and an over-long messages array is rejected with an error, not trimmed
count tokens per call, trigger summarization at a threshold (~80%), keep a structured state object that survives compaction
🪤 Trap: "the long-running support agent suddenly gets API errors after ~40 turns — what happened?" The array outgrew the window, and on the raw Claude API nobody was compacting — there is no automatic anything. The answer is harness-side compaction + trimmed tool results, never "the API should handle it."
The budget mindset
The context window (200K tokens standard on current Claude models; larger tiers exist) is a budget, not a dumpster. Everything competes: system prompt, tool definitions, conversation history, tool results, retrieved documents, thinking tokens (the model's internal reasoning budget — see extended thinking), and the output allowance (max_tokens reserves from the same window).
cache the static prefix (system prompt, tool defs, big docs) so repeat calls skip reprocessing — order matters: stable content first, volatile last; cache_control breakpoints mark the cached prefix; cache lifetime — TTL (time-to-live) ~5 min, extendable — and cache reads cost ~10% of input price
Compaction / summarization
replace old turns with a summary when nearing limits (Claude Code's /compact)
Context isolation via subagents
heavy work in a worker's own window; return summaries
Just-in-time retrieval
fetch data by tool call when needed instead of pre-loading everything
Structured note-taking
agent persists state to files/scratchpads outside the window, re-reads on demand
Long-doc placement
documents at top, instructions after; quote-then-answer for grounding
🎯 Caching questions test order sensitivity: a cache breakpoint only hits if everything before it is byte-identical. Put timestamps, user ids, and dynamic content AFTER the cached prefix — one changing byte upstream invalidates the cache.
🔍 See it run — turn 42 of a support conversation: the budget arithmetic, with numbers
Entry point: a support agent is 42 turns into a messy billing dispute. Naively, the window now holds:
Block (diagram box)
Naive
After management
[System + tool defs — static prefix]
5,500
5,500 — but cached: byte-identical every call, so reads bill at ~10%
[History]
148,000
9,200 — turns 1–35 compacted into a summary + a structured state object
[Tool results]
22,000
3,100 — the harness trims each result to the fields the model needs (ids + amounts, not raw API dumps)
[Retrieved docs]
12,000
2,400 — the billing policy is fetched just-in-time by tool call at turn 41, not pre-loaded at turn 1
[Reserved output]
4,000
4,000 — max_tokens reserves from the same window
Total
~191K — one bad turn from the ceiling
~24K — and the model reasons better, in a cleaner context
The state object that survives compaction is the load-bearing trick:
JSON
{"customer": "c_2214", "issue": "double-charge on 2026-07-30", "amount_cents": 4998,
"actions_taken": ["verified charge", "opened dispute D-102"], "promised": "resolution by Friday"}
Turns 1–35 can be summarized away because everything operational lives here — and the escalation packet, if needed, is built from this object, not from re-reading 42 raw turns.
The caching fine print the exam tests: the prefix only hits cache if everything before the breakpoint is byte-identical. Put a timestamp at the top of the system prompt and you invalidate the cache on every single call — the most expensive one-line mistake in agent economics.
Rolling summarization + a structured state object (customer, issue, actions taken) that survives even when raw turns are compacted; tool results trimmed to the fields the model needs; escalation packets built from the state object, not the raw transcript.
Reliability engineering
Error taxonomy — the harness branches on these
The first three are HTTP status codes the Claude API's endpoint returns instead of a normal response; the rest are conditions your harness detects after a response arrives.
raise limit / continue / fail loudly — never parse as complete
Timeouts
slow generation
client timeouts + streaming to detect stalls early
Tool failure
dependency down
is_error tool_result → model adapts, or harness escalates after N attempts
Schema-invalid output
model drift (a model update quietly changing the output's shape over time)
validate → 1 retry with the error fed back → escalate
🔍 See it run — one night of a 10,000-invoice batch job, four failures, zero data loss
Entry point: the nightly extraction job starts at 1:00 AM. The log tells the story — each incident maps to one taxonomy row:
02:14 — 429 rate limit (another team's backfill started). The client reads retry-after: 8, backs off 8s + jitter, resumes. Throughput drops; nothing fails. Without backoff, retry-storms would have hammered the Claude API and turned one 429 into thousands.
02:31 — stop_reason: max_tokens on invoice #6,201 (a 14-page monster). The harness checks stop_reasonbefore parsing, sees the truncation, re-runs that one document with a higher limit. The naive version parses half a JSON object and either crashes or — worse — silently stores a truncated total.
03:07 — timeout mid-write. Invoice #7,455's request dies after the extraction was recorded but before the client got the response. The retry fires with idempotency key inv-7455 — the effects ledger says "already recorded" → skip. This is why idempotency is a taxonomy answer and not a nicety: retry + side effect without it = duplicate rows at 3 AM.
03:40 — 529 overloaded, persisting. After 3 bounded retries, the degradation ladder engages: remaining invoices route to the fallback model tier; 112 low-confidence extractions get flagged for human review instead of being guessed at. Honest failure beats silent failure — nobody discovers a hole in the data three weeks later.
07:00 — the report: 10,000 processed, 9,888 clean, 112 flagged, 0 lost, 0 duplicated. Cost within budget because the static prefix (instructions + schema) was cached across all 10K calls.
The exam version of this walkthrough is any question ending "…what should the harness have done?" — the answer is always the boring row in the taxonomy table, never heroics.
Resilience patterns
Idempotency: retried requests must not double-execute side effects — idempotency keys + effects ledgers at the tool layer.
Checkpointing: persist agent state per step; a crash resumes at the last checkpoint instead of replaying side effects.
Graceful degradation ladder: primary model → fallback model → cached/templated response → honest failure + human handoff. Never silent failure.
Timeout + kill-tree (kill the tool process and every child it spawned, so nothing lingers) on every subprocess/tool; budgets (turns/tokens/cost/wall-clock) as terminal gates.
Streaming (SSE — Server-Sent Events) for UX and stall detection on long generations.
Batch API for non-interactive workloads (evals, backfills): ~50% cost, hours-scale SLA (service-level agreement — the guaranteed turnaround window) — the exam's answer for "10,000 documents overnight." (Pricing/turnaround as of mid-2026 — check current docs.)
Observability — every action explainable, auditable, replayable
Log per step: prompt version + model + parameters, tool calls with args/results, stop reasons, token usage + cost, gate verdicts, and terminal status — with correlation ids per run (a shared id stamped on every log line of one run, so you can filter to just that run). Traces make agent debugging possible; aggregate metrics (success rate, escalation rate, cost/run, p95 latency — the slowest 5% of runs) make it manageable; evals make change safe (golden sets + negative controls — attack/should-fail cases that must stay failing — run on every prompt/model change).
🪤 Trap: "the agent works in the demo but fails unpredictably in production — first step?" The tested answer is instrument it (tracing/logging of the full loop), not "improve the prompt" or "switch models." Diagnosis before treatment.
Model version pinning & change management
Pin exact model versions in production (claude-…-20260219-style ids, not aliases); upgrades go through the eval suite first; keep a rollback path. Alias tracking is for dev, pinning is for prod — a recurring one-liner question.
Drill-down: rate-limit architecture at scale
Token buckets per model (each model gets a refilling allowance of requests; when it's empty, callers wait); central client with concurrency caps; queue with priority classes (interactive > batch); shed or defer batch work under pressure; per-tenant quotas so one customer can't starve others; monitor input/output token rates, not just request counts.
Drill-down: cost engineering checklist
Cache the static prefix (largest lever in agent loops).
Tier models per role (cheap classify / frontier reason / a judge model from a different family, so it doesn't rubber-stamp its own kind's output).
Trim tool results — return ids + summaries, not payload dumps.
Batch API for offline work; streaming for interactive.
Cap loops (max turns) and alert on cost-per-run outliers.
Track tokens as a first-class metric per feature.
Context & reliability as real files — annotated, clickable
JSONC
// the MOVING BREAKPOINT — this turn WRITES the cache, the next turn READS it
{
"system": [{ "type": "text", "text": "…8K of stable rules…",
"cache_control": { "type": "ephemeral" } }], // fixed breakpoint
"messages": [ /* …history… */,
{ "role": "user", "content": [{ "type": "text", "text": "newest question",
"cache_control": { "type": "ephemeral" } }] } // moving breakpoint
]
}
JSONC
// a COMPACTION trace event — a routine, logged decision, never an outage
{ "type": "compaction", "before_tokens": 161240, "after_tokens": 42730,
"kept_turns": 8, "facts_saved_to_state": ["migrated_files", "open_ids"] }
PYTHON
# PRE-FLIGHT fit check — measure, don't guess (count_tokens is exact, free, no generation)
n = client.messages.count_tokens(model=MODEL, system=system, tools=tools, messages=msgs).input_tokens
if n + MAX_OUT + MARGIN > WINDOW:
msgs = compact(msgs) # BEFORE create — never learn "too long" from a 400
Specimens in the wild: cookbook:misc/prompt_caching.ipynb · cookbook:misc/session_memory_compaction.ipynb — client-side compaction + the moving breakpoint · cookbook:tool_use/automatic-context-compaction.ipynb — server-side, threshold-driven · cookbook:cost_optimization/cost_optimization.ipynb.
Named reliability patterns the exam tests
Lost in the middle — position effects
Models process the beginning and end of long inputs reliably and may drop findings from the middle. Two tested mitigations: place a key-findings summary at the top of aggregated input, and organize detail under explicit section headers so nothing important lives only mid-document. Related budget leak: verbose tool results (a 40-field order lookup when 5 fields matter) — trim to return-relevant fields before they accumulate, and extract transactional facts (amounts, dates, order numbers) into a persistent "case facts" block that rides outside the summarized history, where progressive summarization can't blur them into vagueness.
Scratchpads & crash-recovery manifests
Extended sessions degrade: the model starts answering from "typical patterns" instead of the specific classes it discovered an hour ago. Two structural fixes:
Scratchpad files — agents write key findings to a file and reference it for later questions, persisting knowledge across context boundaries.
Crash-recovery manifests — each agent exports structured state to a known location; on resume, the coordinator loads the manifest and injects it into agent prompts. Recovery is a design artifact, not an accident of context survival.
Delegate verbose exploration to subagents; summarize each phase before spawning the next; /compact when discovery output fills the window.
Aggregate accuracy (a proud "97% overall") can mask poor performance on specific document types or fields — validate accuracy per segment before reducing human review anywhere.
Stratified random sampling of high-confidence extractions catches novel error patterns that confidence scores miss.
Field-level confidence scores are only usable after calibration against labeled validation sets — raw self-reported confidence is poorly calibrated (the same reason self-reported confidence is a poor escalation trigger).
Route to humans: low-confidence extractions and ambiguous/contradictory source documents — spending limited reviewer capacity where the model is least trustworthy.
Provenance — claim-source mappings through synthesis
Summarization steps lose attribution unless the pipeline forces structure: subagents output claim → source mappings (URL, document name, relevant excerpt) that synthesis agents must preserve and merge, never flatten. Three tested corollaries:
Conflicting statistics from credible sources → annotate the conflict with both attributions; never arbitrarily pick one.
Temporal data → require publication/collection dates in structured outputs so time differences aren't misread as contradictions.
Coverage honesty → synthesis output carries annotations distinguishing well-supported findings from topic areas with gaps (sources unavailable), and renders content natively — financial data as tables, news as prose, technical findings as lists — instead of one uniform format.
Practice this domain
🎯 Done with Domain 5? Prove it: Context & Reliability (Q27–Q30) — commit to an answer before revealing. Locally, your picks are captured with timestamps and scored per section, so weak spots surface on the practice page's comfort tiles.