Domain 5 – Facts

Think of the window as one bucket. The system prompt, tool definitions, history, tool results, retrieved documents, thinking tokens and the reserved output all come out of it. max_tokens is not a hint; it is space the API sets aside, so if input plus that reservation is too big, the API rejects the request before it generates anything. That is why "just raise max_tokens" makes a fit problem worse, not better.

The count endpoint tokenizes the real system, tools and messages you are about to send, so it also sees a changed tool list or a big new tool result. The last response's usage only tells you what the *last* call cost, and chars/4 is a rough guess that is often off on code, JSON and non-English text, so together they can miss by thousands of tokens. Waiting for the context-overflow 400 wastes a round trip, and in an agent loop it may come after side effects already ran. The count is still an estimate, so keep a safety margin.

Old tool results are the cheapest thing to cut: a 40-field lookup from ten turns ago costs nothing to drop, and the model no longer needs it. Summarizing history loses detail, so it comes second, and you keep exact state outside the summary. Shrinking the output reservation is a real trade-off: too small, and the answer gets cut off with stop_reason: "max_tokens". Raising max_tokens takes more of the same window, so it can only make the overflow worse.

Turn count is a bad stand-in for size, because tokens arrive unevenly: one big document or tool result can outweigh twenty short messages. Counting the full request (instructions, tools, messages, results) before each call catches the spike right away. The math (window minus reserved output minus safety margin, trigger at some percent) is a choice your app makes; the API has no built-in 75% or 80% rule. If you expect a big tool result, compact earlier, or store the full result and pass only a small excerpt.

stop_reason is the API telling you *why* the model stopped. "max_tokens" means it ran out of reserved output space and stopped in the middle, maybe in the middle of a JSON object. A careless harness (your code that runs the loop) parses whatever came back and crashes, or worse, saves a half-finished object as if it were real. So check the reason first: re-run with a bigger limit, continue the output, or fail loudly so a person sees it.

Models pay good attention to the start and end of a long input and can under-use the middle, even when every token is there. The logs are the clue: the middle content was sent, so nothing was cut by compaction, trimmed by the API or removed by caching (caching reuses a prefix; it never deletes content). Help the model by putting key findings first, adding section headers, splitting by section, or running focused extraction passes over the middle part.

Tool results pile up in context and most of it is noise: if a lookup returns 40 fields and the next step uses five, 35 useless fields ride along on every turn. Filter when you add the result, so the junk never enters the window. A bigger window tier only delays the problem, costs more, and the extra fields still distract the model; compacting later loses detail and reacts too late. If you need the full payload for audits, keep it in storage and pass a reference.

Nothing falls out of a Messages API conversation by itself: old turns stay in the array until your code summarizes or removes them, so a long raw-API agent eventually sends a request bigger than the window and gets an error. Claude Code compacts as a product feature, and the Agent SDK inherits that engine; the Messages SDK tool runner and the raw API do only what you set up. For a raw-API agent the fix is an explicit policy: count tokens each call, summarize or enable server compaction, and keep a state object that survives the summary.

A beta feature is switched on by naming it in the anthropic-beta header (the SDK fills it from the betas list). What the feature should *do* is set separately in the request body under context_management. The header name uses dashes, the body type uses underscores, and both carry the date from the docs, not today's date. Copy both strings exactly from the current docs; client.beta by itself turns nothing on.

The notebook used a tiny 5,000-token trigger just so the summary would fire in a short demo; the real minimum is ten times higher, so a real request can't use that value. The notebook's compaction_control argument came from older Python runner code and is gone in the v1 SDK; use context_management.edits instead. When the API compacts, it returns a compaction block inside content. Keep that block in the assistant message you send back, or the next turn loses the summary.

App-managed summarizing is just one more model call: your code picks the older finished exchanges, sends them with "summarize this" instructions, and reads the text that comes back. It costs tokens like any call, and there is no special helper. Before you swap out history, check the summary finished (stop_reason == "end_turn") and has real text. If it was a refusal or got cut off, keep the original checkpoint, or you trade real history for nothing.

Every tool_use block needs a matching tool_result in the next user message, so cutting between them gives the API a broken message sequence. Pick a cut point after a complete exchange, and if the model asked for several tools at once, include all their results. Summarizing changes only the prompt: the run id, the turn counter and the ledger (record) of actions already done are your app's state. Reset them and your budgets and duplicate-action checks quietly start over.

A prose summary reads well but drops details: "the customer was unhappy about a charge" can lose the promised refund amount or the dispute id that later decides what the agent may do. A structured object with named fields is carried forward by your app and updated from verified results, so it survives every compaction. Transcript search and subagent isolation help with size, but they don't keep those facts in front of the model. Build the escalation packet from this object, not by re-reading raw turns.

The request log is your evidence: if the promise isn't in the messages the later call got, the model never saw it, so you don't need any attention theory. Attention problems only apply when the information was there but under-used. A model's context capacity doesn't shrink as turns pile up; used space grows, capacity stays the same. Prompt caching reuses a prefix and removes nothing. Fix the harness: keep commitments in verified structured state next to the summary.

Each summary of a summary loses precision: a specific rename, or an agreed rule like "no synchronous calls in the event-handler layer", becomes "various refactoring changes", and the model then works from an old fact it still remembers. Keep renames, rules and their reasons in a durable facts block or scratchpad outside the summarized history, and re-read the current source before the final output. A stale-cache story doesn't fit: Claude Code reads the current file every time.

After dozens of tool calls the window is full of long discovery output, and the model starts answering from "typical patterns" instead of the exact classes and signatures it found earlier. That comes from tokens piling up, not from a turn limit, a safety filter or a taste for short answers. Another /compact squeezes harder but can't bring back details already blurred, and temperature changes randomness, not memory. Save key findings to a scratchpad file the agent re-reads, and hand the rest of the noisy exploration to subagents that return short summaries.

The point of a subagent is isolation: the search results and file dumps of discovery stay in the worker's window, and only a short structured summary comes back, so the build phase starts clean. It does not get a bigger window, more permissions or automatic parallelism; you only get parallel speed if you start several workers on purpose. Between phases, pass a short brief (interfaces, conventions, file locations, constraints), not raw output. Lowering max_tokens makes answers shorter but removes nothing already in context.

The cache matches exact bytes, not meaning: the API reuses the processed prefix only when every byte before the breakpoint matches the earlier request. A timestamp at the top changes the first bytes each call, so everything behind it misses and cache_read_input_tokens stays at zero. Move changing values after the breakpoint and check hits in usage. The other rules still apply too: minimum cacheable length, TTL (how long the cache lives), and same model.

Put the parts that never change at the front (system prompt, tool definitions, large reference docs) and the parts that change every call after the breakpoint (timestamps, user ids, the new turn). The part before the breakpoint is what gets reused; anything added after it is processed fresh and sits next to the cached prefix. Thinking new content after the marker breaks the earlier block has it backwards: only changes *before* the marker break the match.

cache_control belongs to a content block, so system has to be a list of blocks for the marker to sit inside the block you want cached; a plain string system has nowhere to put it. A top-level cache_control on the request is a different thing: it turns on automatic caching instead of marking one breakpoint. You can set caching up from the very first request: that call writes the cache and later calls read it.

A cache hit reuses the already-processed prefix, so those tokens cost a small part of the normal input price. The cache lives a few minutes by default, each hit refreshes it, and a longer TTL is available. The usage block tells you what happened: cache_creation_input_tokens when it writes, cache_read_input_tokens when it hits; a read count of zero means nothing was reused. Caching cuts cost and reprocessing only; cached tokens still count toward the window, so it never fixes a fit problem.

In an agent loop the same 30K of instructions and tool definitions go out every turn, so that repeated part is most of the bill; caching turns it into cheap cache reads. A smaller model changes quality and needs re-testing, fewer iterations may skip needed work, and shorter tool results only trim the small changing tail. None of them hits the part that repeats. Put stable content first, mark the breakpoint, and confirm the saving in usage.

When your app handles caching itself, it puts cache_control on the newest eligible message each turn. That call writes a cache covering everything up to there, and the next call, whose prefix now includes that message, reads it, so the reused part grows with the chat. Marking the oldest message would cache a prefix that never grows; the API does not move markers by itself in this design; and the marker adds no memory, it only saves reprocessing.

When no one waits for an individual result, async batching fits: send requests each with a custom_id, poll until done, and read one result line per item to match it back to its document. Batches cost less and finish within a documented window, but that window is a planning number, not a promise; an overnight cutoff still needs a schedule, retries for failed items and a fallback. Firing real-time calls as fast as possible just adds rate-limit pressure and gains you nothing on latency.

"succeeded" is the API's verdict on the request, not on your data: the model may have pulled the wrong total from a perfectly good call, so match each result by custom_id and validate it before saving. Batches are also not a general scheduler. Each item is one separate request, so a multi-turn agent workflow can't run inside one, and queuing work for hours doesn't bring back a service that is down.

Server-Sent Events send tokens as they are generated, so the user sees output right away and your client can notice a stuck generation early instead of waiting for a timeout. That fixes interactive waiting and long-answer monitoring, nothing else. A backlog of 10,000 documents needs scheduling and per-item tracking, and steps that depend on each other stay in order whether they stream or not.

The status code tells your harness which path to take. 429 means you sent too much, so wait with jitter (a bit of random delay) and respect any retry-after. 529 means the service itself is full, so wait, and switch models only if it lasts. 500 is a short-lived fault worth a few retries. These come back *instead of* a response, so they have nothing to do with stop_reason, which explains why a successful answer ended. Retrying with no wait turns one rate limit into a retry storm.

retry-after is the server telling you exactly how long to wait, so it beats any backoff formula; it comes as a response header, not inside the rate_limit_error JSON body. Without it, exponential backoff with jitter spreads retries so clients don't all retry at once. Under long pressure, drop or delay batch work so live traffic keeps flowing. Switching models first ignores the wait you were given and changes output quality for what may be a ten-second blip; a fallback makes sense only when overload lasts.

A 403 means the caller isn't allowed; waiting doesn't change permissions, so retries waste budget and delay the real fix. What matters is that the *type* of failure reaches the coordinator: "access denied to this source" lets it pick another source or report a coverage gap. The worst case is a subagent that catches the error and returns an empty result marked success, so the report silently leaves out a whole kind of evidence.

A search that finished and found nothing is a real finding: the source was checked and has no matches, and the report should say so as a coverage note. A connection timeout is different: the call never ran, so the result is unknown and a limited retry makes sense. Retrying both three times wastes calls on the finished search and hides what actually happened. A broader search is a *new* query, not a retry of a failed one.

A subagent's timeout is information the coordinator needs: what failed, what was tried, what partial results exist, and what options remain. With that it can retry with a narrower query, switch sources, or go on with a documented gap. A vague "search unavailable" hides those options, an empty result marked success hides the failure completely, and stopping the whole workflow throws away every other subagent's finished work.

If one of three evidence sources was down, a polished report that reads as complete is misleading about what it's based on. Marking findings as well supported versus gap areas lets the reader see what the evidence really covers. Filling gaps from the model's training knowledge (called parametric knowledge), even labeled "model-estimated", adds unsupported claims and mixes up where things came from; stopping the run throws away useful partial results.

A timeout in the middle of a write leaves the outcome unknown (the row may or may not exist), and retrying is the right move; the bug is that the write tool did the same thing twice. An idempotency key (for example the invoice id) plus an effects ledger (a record of writes already done) lets the second try see the work is done and skip it. Backoff only changes *when* the retry fires, and checkpointing restarts the run at the right step but doesn't make one write safe to repeat.

Each step gives up a little quality to keep working: a fallback model, then a cached or template answer, then a clear failure handed to a person, and someone can see every step. Silent failure is the one thing not allowed, because nobody finds the hole in the data until weeks later. In a batch job, after a limited number of overload retries the ladder kicks in: remaining items go to the fallback tier, and low-confidence outputs are flagged for review instead of guessed.

If a dependency times out, the tool should return a tool_result with is_error: true and a useful message, so the model can apologize, retry or work around it; an uncaught exception just kills the loop before the model learns anything. is_error is information, not a stop sign: your app's policy still decides whether to retry, continue or escalate after N tries. Schema-invalid output gets the same treatment: validate, send the error back for one retry, then escalate instead of looping forever.

A production-only failure could be in the prompt, a tool, a backend, the parser, or an ignored stop reason; without a trace you are guessing. Full-loop tracing records each run's prompts, tool calls and results, stop reasons and checks under one correlation id (one shared id per run), so you can find the first wrong step. Retries everywhere can repeat side effects and hide the cause, and changing prompts or models changes the system before you know what's broken. Find the cause first, then fix.

A useful trace records what you sent (prompt version, model id, parameters), what came back (stop_reason, token usage, cost), what your harness did with it (tool args and results, check results) and how the run ended, all with the same correlation id so you can pull one run out of thousands. Per-run traces make debugging possible; totals like success rate, escalation rate, cost per run and p95 latency show drift before customers notice. Handle sensitive data in traces on purpose.

An alias points to whatever model version is current, so production on an alias can change behavior overnight without any deploy. A dated model id names one version and changes only when you change it. Upgrades then become a choice: run the new model through your eval suite, compare with the pinned one, and keep the old id to roll back. Aliases are fine in development, where you want the latest.

The golden set is your fixed baseline: checked inputs with known-correct outputs, run the same way each time, so a score change can only mean the prompt or model changed. Negative controls (inputs that must keep failing or being refused) catch regressions the other way. If the model under test writes the expected outputs, the eval compares the model with itself and hides every regression; rotating production samples help spot drift but keep moving the baseline. The set can change only through explicit, versioned review, never quietly.

A confidence threshold was tuned on the documents seen so far; a new two-column layout can change how score relates to correctness, so a reported 92% may not mean 92% accurate. Regularly sampling auto-approved, high-confidence extractions across every document type and field (stratified = drawn from each group), with people labeling them, measures the real error rate where no labels exist and shows new failure patterns early. A format classifier can miss a new layout (fine as an extra check, not a replacement), a higher global threshold hurts well-understood types, and an old labeled set never sees the new documents.

A model saying "0.95" doesn't prove 95% accuracy until its scores are compared with labeled results; calibration maps raw scores to real accuracy, field by field, because dates and totals behave differently from penalty clauses. Once calibrated, send only the truly uncertain fields to reviewers, so their limited time goes where errors happen. Whole documents flood reviewers with fields that are fine, a flat 10% sample misses where errors cluster, and trusting uncalibrated self-scores repeats the exact problem calibration fixes.

A weighted average follows the high-volume type: 99% on invoices and 97% on purchase orders can lift 68% on contracts to a proud 95% overall, and contracts are the legally sensitive ones. Before automating on confidence, measure accuracy by document type and field, send weak types to mandatory review until they improve, and keep sampling auto-approved output after that. A higher global threshold doesn't help when the model is confidently wrong on the weak type, and business-rule alerts catch only errors that break a rule.

Sources survive a pipeline only if they travel as data: each claim carries ids that point to a source record (URL or document, location, retrieval date), and the synthesis step merges those mappings instead of flattening everything into prose. If earlier findings are summarized without the mappings, the citations are gone before synthesis starts, however clearly the synthesis prompt asks for them. A code bug that reads only the main content field is worth checking in the logs, but the real lesson is to make sources part of the structure so no step can drop them.

A sentence like "according to page 12 of the annual report" is lost as soon as another system pulls the number out of the text; a record with value, source and page fields moves and is stored as one unit, so the source stays attached. For generated documentation, provenance means each claim links to the source file, line range and retrieval time that supported it, so after a refactor reviewers re-check exactly those files. Git history shows when the doc was written, not which sources back a statement, and a separate attribution database adds infrastructure for something the extraction can output directly.

Two credible numbers usually differ for a reason: a different year, sample size, method or peer-review status, and that reason is what the reader needs. Show both values with their source and date so an expert can weigh them; averaging makes up a number neither source reported, and picking the newest quietly throws away valid context. When the dates alone explain the gap, the numbers don't actually conflict, so a note is enough and no person needs to step in.