Domain 5 – Questions

Q27. An agent loop resends a 30K-token system prompt + tool defs every iteration. Biggest cost lever: A) Smaller model B) Fewer iterations C) Shorter tool results D) Prompt caching the static prefix (stable content first, dynamic after the breakpoint)

{
  "model": "claude-sonnet-4-6",
  "max_tokens": 1024,
  "system": [
    {
      "type": "text",
      "text": "Summarize documents using only supplied facts.",
      "cache_control": {
        "type": "ephemeral",
        "ttl": "5m"
      }
    }
  ],
  "metadata": {
    "user_id": "user_demo_opaque_104"
  },
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "document",
          "source": {
            "type": "text",
            "media_type": "text/plain",
            "data": "Order O-104 shipped on Monday."
          },
          "title": "Order record",
          "citations": {
            "enabled": true
          }
        },
        {
          "type": "text",
          "text": "When did this order ship? Cite the record."
        }
      ]
    }
  ]
}

Q28. The cache never hits. Most likely cause: A) TTL expired instantly B) Wrong model C) A timestamp/dynamic value sits before the cache breakpoint, changing the prefix bytes D) Caching is off in prod

{
  "model": "claude-sonnet-4-6",
  "max_tokens": 1024,
  "system": [
    {
      "type": "text",
      "text": "Summarize documents using only supplied facts.",
      "cache_control": {
        "type": "ephemeral",
        "ttl": "5m"
      }
    }
  ],
  "metadata": {
    "user_id": "user_demo_opaque_104"
  },
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "document",
          "source": {
            "type": "text",
            "media_type": "text/plain",
            "data": "Order O-104 shipped on Monday."
          },
          "title": "Order record",
          "citations": {
            "enabled": true
          }
        },
        {
          "type": "text",
          "text": "When did this order ship? Cite the record."
        }
      ]
    }
  ]
}

Q29. 10,000 documents need extraction with enough time for the provider’s batch completion window and retries; no user is waiting on an individual result. Use: A) Parallel real-time calls at max rate B) The Batch API with per-item tracking and a deadline/retry plan C) Streaming D) One giant prompt

{
  "requests": [
    {
      "custom_id": "invoice-104",
      "params": {
        "model": "claude-sonnet-4-6",
        "max_tokens": 512,
        "messages": [
          {
            "role": "user",
            "content": "Extract invoice INV-104, total USD 12.50. Return the amount in cents."
          }
        ],
        "output_config": {
          "format": {
            "type": "json_schema",
            "schema": {
              "type": "object",
              "properties": {
                "invoice_number": {
                  "type": "string"
                },
                "total_cents": {
                  "type": "integer"
                }
              },
              "required": [
                "invoice_number",
                "total_cents"
              ],
              "additionalProperties": false
            }
          }
        }
      }
    }
  ]
}

{
  "custom_id": "invoice-104",
  "result": {
    "type": "succeeded",
    "message": {
      "id": "msg_demo_3",
      "type": "message",
      "role": "assistant",
      "model": "claude-sonnet-4-6",
      "content": [
        {
          "type": "text",
          "text": "{\"invoice_number\":\"INV-104\",\"total_cents\":1250}"
        }
      ],
      "stop_reason": "end_turn",
      "stop_sequence": null,
      "usage": {
        "input_tokens": 180,
        "output_tokens": 35,
        "cache_creation_input_tokens": 0,
        "cache_read_input_tokens": 0
      }
    }
  }
}

Q30. Production incident: the agent "works in staging, fails weirdly in prod." First architectural step: A) Full-loop observability — trace prompts, tool calls, stop reasons, gates per run — then diagnose B) Add retries everywhere C) Rewrite prompts D) Swap models

Q71. A team enables prompt caching with a cache_control breakpoint after an 8K system prompt, then adds a "Generated at: {timestamp}" line at the top of that prompt. Every call now reports cache_read_input_tokens: 0. Why? A) Every call misses because the cached prefix is too short to reach the minimum cacheable length B) Adding any text after the breakpoint necessarily invalidates every earlier cached block C) Every call misses because the prefix before the breakpoint is no longer byte-identical between calls D) Every call misses because the prefix before the breakpoint is no longer semantically identical between calls

{
  "model": "claude-sonnet-4-6",
  "max_tokens": 1024,
  "system": [
    {
      "type": "text",
      "text": "Summarize documents using only supplied facts.",
      "cache_control": {
        "type": "ephemeral",
        "ttl": "5m"
      }
    }
  ],
  "metadata": {
    "user_id": "user_demo_opaque_104"
  },
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "document",
          "source": {
            "type": "text",
            "media_type": "text/plain",
            "data": "Order O-104 shipped on Monday."
          },
          "title": "Order record",
          "citations": {
            "enabled": true
          }
        },
        {
          "type": "text",
          "text": "When did this order ship? Cite the record."
        }
      ]
    }
  ]
}

Q72. A raw-API support agent promised a refund at turn 6. At turn 41, the request log confirms that summarization removed the promise from the supplied history, and the agent denies it. What happened? A) The model forgot the promise because its attention decayed across 41 turns B) The model forgot the promise because the context window shrank as history grew C) The cache removed the promise from an otherwise complete request D) The harness dropped the promise when it summarized turns 1–35, so the model never saw it

{
  "model": "claude-sonnet-4-6",
  "max_tokens": 4096,
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Help me plan a long research task."
        }
      ]
    }
  ],
  "context_management": {
    "edits": [
      {
        "type": "compact_20260112",
        "trigger": {
          "type": "input_tokens",
          "value": 100000
        }
      }
    ]
  }
}

Q73. Before each call, a raw-API harness must verify input + max_tokens will fit the context window, and trigger its configured compaction policy if needed. Every option below yields a usable number — which token counting approach is best? A) Call the count_tokens endpoint on the exact system + tools + messages payload before create B) Take usage.input_tokens from the previous response and add a chars/4 estimate of the new turn C) Estimate the whole array at roughly 4 characters per token with a safety margin D) Send the request and treat a context-overflow 400 as the signal to compact and retry

{
  "model": "claude-sonnet-4-6",
  "system": "You help customers check order status. Use get_order for order facts. Never invent a status.",
  "tools": [
    {
      "name": "get_order",
      "description": "Read the status of an order the signed-in customer may access.",
      "input_schema": {
        "type": "object",
        "properties": {
          "order_id": {
            "type": "string",
            "description": "The order identifier, for example O-104."
          }
        },
        "required": [
          "order_id"
        ],
        "additionalProperties": false
      },
      "strict": true
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Where is order O-104?"
        }
      ]
    }
  ]
}

{
  "input_tokens": 180
}

Q74. A billing-dispute agent will run 60+ turns on the raw API. These context-management approaches can be useful — which most directly addresses the requirement for keeping the dispute's operational facts reliable to the end? A) Trigger summarization at ~80% of the window and carry a rolling prose summary in place of old turns B) Trigger summarization at ~80% and additionally keep a structured state object (customer id, amounts, actions taken, promises) that survives every compaction C) Skip summarization; fetch prior turns just-in-time through a transcript-search tool when the model asks D) Isolate each turn's heavy tool work in subagents so the main window barely grows

{
  "model": "claude-sonnet-4-6",
  "max_tokens": 4096,
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Help me plan a long research task."
        }
      ]
    }
  ],
  "context_management": {
    "edits": [
      {
        "type": "compact_20260112",
        "trigger": {
          "type": "input_tokens",
          "value": 100000
        }
      }
    ]
  }
}

Q75. A raw-API agent's pre-flight token counting shows input + max_tokens about to exceed the context window. What is the correct order of levers? A) Summarize history first, then drop stale tool results, then raise max_tokens to fit more B) Raise max_tokens first for headroom, then summarize history, then drop tool results C) Drop stale tool results first, then summarize history, then raise max_tokens if both fall short D) Drop stale tool results first, then summarize history, then shrink the reserved output only if both fall short

{
  "model": "claude-sonnet-4-6",
  "max_tokens": 4096,
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Help me plan a long research task."
        }
      ]
    }
  ],
  "context_management": {
    "edits": [
      {
        "type": "compact_20260112",
        "trigger": {
          "type": "input_tokens",
          "value": 100000
        }
      }
    ]
  }
}

Q76. At peak traffic your client starts receiving 429 responses, and occasionally 529. What is the correct precedence of responses? A) Retry immediately once, then fall back to exponential backoff with jitter, then queue B) Honor the retry-after header when present, else exponential backoff with jitter; shed or defer batch-priority load; consider a fallback model only for persistent 529 C) Switch to the fallback model first to relieve pressure, then backoff, then honor retry-after D) Queue all traffic first, then backoff with jitter, then honor retry-after on the drained queue

{
  "type": "error",
  "error": {
    "type": "rate_limit_error",
    "message": "Request rate limit exceeded."
  },
  "request_id": "req_demo_429"
}

Q77. In prompt caching for multi-turn agents, in an application that explicitly updates cache_control each turn, what is a “moving breakpoint”? Not on CCAR-F A) A breakpoint the API relocates automatically as the conversation grows past the cache TTL B) A cache_control marker on the system prompt that shifts earlier each turn so the cached prefix shrinks C) A cache_control marker placed on the newest message each turn, so each call writes the cache that the next call reads D) A cache_control marker placed on the oldest message each turn, so each call writes the cache that the next call reads

{
  "model": "claude-sonnet-4-6",
  "max_tokens": 1024,
  "system": [
    {
      "type": "text",
      "text": "Summarize documents using only supplied facts.",
      "cache_control": {
        "type": "ephemeral",
        "ttl": "5m"
      }
    }
  ],
  "metadata": {
    "user_id": "user_demo_opaque_104"
  },
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "document",
          "source": {
            "type": "text",
            "media_type": "text/plain",
            "data": "Order O-104 shipped on Monday."
          },
          "title": "Order record",
          "citations": {
            "enabled": true
          }
        },
        {
          "type": "text",
          "text": "When did this order ship? Cite the record."
        }
      ]
    }
  ]
}

Q78. Your eval suite gates every prompt and model change. What precisely is the "golden set"? A) A fixed, vetted set of inputs with known-correct expected outputs, re-run unchanged on every prompt or model change to detect regressions B) A rotating sample of recent production traffic, re-labeled weekly so the eval tracks drift C) A fixed, vetted set of inputs with expected outputs generated fresh by the model before each release D) The highest-scoring production runs, kept as exemplars for few-shot prompting

Q79. Nightly batch extraction: a handful of invoices appear twice in the database, and every duplicated invoice corresponds to a request that timed out mid-flight. Root cause? A) The Batch API replayed those requests server-side after the timeout B) Exponential backoff was missing, so the retries fired too quickly after the timeout C) The client correctly retried after the timeout, but the write tool had no idempotency key or effects ledger, so the replay double-executed the side effect D) Checkpointing was missing, so the resumed run restarted from the wrong step and re-processed those invoices

# Offline teaching simulation: no payment or shipment is performed.
# Production must make effect recording atomic with the operation, or use
# the provider's idempotency support and reconcile uncertain outcomes.
ledger = {}

def propose(operation_id, amount, approved=False):
    if operation_id in ledger:
        return ledger[operation_id]
    if amount > 50 and not approved:
        return {"status": "pending_approval"}
    result = {"status": "completed", "operation_id": operation_id}
    ledger[operation_id] = result
    return result

print(propose("credit-104", 75))
print(propose("credit-104", 75, approved=True))
print(propose("credit-104", 75, approved=True))

Q80. A synthesis agent aggregates twelve source documents into one long input. It reliably reports findings from the first and last sources but keeps missing findings that appear only in the middle — yet request logs confirm the middle sources were present, and the total was well inside the context window. Root cause? A) The harness silently dropped the middle sources during compaction before the call B) Position effects — "lost in the middle": the model attends reliably to the beginning and end of long input and can drop findings from the middle C) The middle sources pushed the array past the window, so the API trimmed them D) A prompt-caching breakpoint truncated the middle of the cached prefix

Q318. A platform engineer has Claude Code migrating 40 microservices from a homegrown logger to structured logging. For the first dozen services its notes named the exact wrapper functions it replaced, such as logCtx() and emitAudit(). By service 26 the notes read "this service uses the usual logger setup," and the session has already been compacted twice. Fourteen services remain. Which plan restores precise work for those fourteen? A) Write the replacements found so far to a migration notes file, then hand each remaining service to a subagent that reads that file and returns a short structured list of the functions it changed. B) Clear the conversation and have one fresh session rescan all 40 services from the beginning so the rebuilt context contains only the essential names. C) Run the compaction a third time with a focus instruction to preserve every function name, then carry on with the remaining services in the same session. D) Add a line to the prompt telling the agent to always cite exact function names and file paths, and resume the migration in the current session.

Q319. An agent generates migration scripts for 30 warehouse database tables, one table at a time. After each table it summarises the schema in a sentence or two and drops the raw DDL to save room. While handling table 9 it saw that the column cust_ref had been renamed to customer_id everywhere. The script it wrote for table 24 still joins on cust_ref. Which design change would have prevented this? A) Keep a small ledger of cross-table facts, such as renames and type changes, outside the summaries and include it unchanged in every step. B) Switch to a model with a larger context window so the raw DDL for every table fits and nothing has to be summarised at all. C) Lower the sampling temperature for the script-writing step so the agent reproduces column names more consistently across tables. D) Before each new table, have the agent reread its own summary of table 9 so the earlier change stays fresh in its working context.

Q320. A security team used an agent to condense 200 internal policy documents into a two-page answer for a customer's vendor questionnaire. One line reads "customer data is encrypted at rest with customer-managed keys." A year later an auditor asks which policy supports that line and whether it still applies after the key-management redesign. Which output design would have made that question a quick lookup? A) Run a second model pass over the answer before sending it to confirm each sentence against the policy set at that time. B) Attach a bibliography at the end of the answer listing all 200 policy documents that were given to the agent as input. C) Store each claim as a record naming the source document, section, quoted excerpt and retrieval date alongside the sentence. D) Save every generated answer in a document store with version history so each line can be traced to the date it was produced.

{
  "claim": "Customer data is encrypted at rest with customer-managed keys.",
  "sources": [
    {
      "doc_id": "POL-KMS-004",
      "section": "3.2 Key custody",
      "excerpt": "Tenant data stores use keys held in the customer key vault...",
      "retrieved_at": "2025-08-14"
    }
  ]
}

Q321. A trip-planning coordinator sends a hotel-availability subagent to check five cities. After finishing three cities, the subagent's upstream booking service starts returning rate-limit responses with a retry-after header. What should the subagent send back so the coordinator can make the best recovery decision? A) Throw the rate-limit exception up to the coordinator's top-level handler, which ends the whole trip plan so that no partial itinerary is produced. B) Return an error result stating a rate-limit failure, the three completed cities with results, the two pending cities, and the retry-after value. C) Retry internally with backoff until its budget is spent, then return one short message stating that hotel search is currently unavailable. D) Return the three finished cities as a normal success result so the itinerary can be built, leaving the other two cities out of the reply.

{
  "status": "partial_failure",
  "error_type": "rate_limited",
  "retryable": true,
  "retry_after_seconds": 30,
  "completed": {
    "Lisbon": [
      "Hotel Almada"
    ],
    "Porto": [
      "Casa Ribeira"
    ],
    "Madrid": [
      "Hostal Sol"
    ]
  },
  "pending": [
    "Seville",
    "Valencia"
  ],
  "alternatives": [
    "retry after delay",
    "query secondary provider for pending cities"
  ]
}

Q322. A research assistant is preparing evidence for a city infrastructure committee on a concrete additive. A peer-reviewed 2019 lab study on 40 test slabs reports a 30% reduction in cracking. A 2025 white paper funded by the manufacturer, covering 900 slabs in real roads, reports 8%. Which statement describes the synthesis agent's job exactly? A) Present both results, each with its source, date, review status, funding, sample size and setting, and note what may explain the gap. B) Weight the two figures by their sample sizes and report the combined estimate of roughly 9% as the best overall effect. C) Treat the peer-reviewed lab study as authoritative because it passed review, and mention the white paper only as an unreviewed claim. D) Leave both results out because they disagree, and tell the committee that no conclusion is possible until a third study is published.

Q323. A labor-market dashboard agent finds that 34% of tech job postings were fully remote in a survey fielded in early 2022, and 22% in a different survey from late 2025. The current design marks every such mismatch as a contradiction and holds the report until a human analyst resolves it. What is wrong with this design? A) Nothing important; escalating every mismatch to an analyst is the safest way to keep contradictory figures out of the published dashboard. B) The design should resolve such mismatches automatically by keeping the most recent figure, since it reflects current market conditions. C) The design should publish the midpoint of the two figures, 28%, with a note that the underlying surveys disagreed somewhat. D) The figures come from different periods, so they may describe a trend, and should be shown together with their sources and dates.

Q324. A pull-request review pipeline runs a license-compliance subagent alongside style and security checks. The license scanner's API token had expired, so its calls returned 401 Unauthorized. The subagent caught the error and returned {"violations": []}, and the pipeline approved a PR that added a library under an incompatible license. Where is the defect? A) In the subagent's retry policy, which should have retried the 401 with exponential backoff before giving up and returning results. B) In the orchestration design, which should halt all review for every repository as soon as any single check fails for any reason. C) In the pipeline, which should refuse to approve any PR unless every check subagent reports at least one finding for a reviewer to look at. D) In the subagent, which turned an authorization failure into a clean-looking empty result, so nothing upstream knew the scan never ran.

Q325. A recruiting agent runs two subagents. One searches the internal candidate database for "Rust and FPGA experience, based in Lisbon"; the query completes normally and returns zero candidates. The other asks the interviewer-calendar service for open slots and receives a 503 Service Unavailable. The harness currently treats both as "no data" and retries each three times. Which handling rule is right? A) Treat both as valid empty answers and move on, reporting that there are no suitable candidates and no open interview slots. B) Retry both with a longer exponential backoff, since either could succeed on a later attempt if the systems are given more time. C) Accept the zero-candidate result as a real answer and note it, and retry the calendar call as a transient service failure. D) Send both straight to the recruiting operations team for manual investigation and remove the automatic retries entirely.

Q326. An insurer auto-approves repair estimates from photo analysis when model confidence is at least 0.88, a threshold calibrated a year ago on photos of sedans and SUVs. A new fleet client sends photos of electric cargo vans taken with fisheye dashcams, and the estimates come back at 0.93 confidence. Which statement is exactly right? A) Raising the threshold for every vehicle type to 0.97 is the right response, since that removes most of the risk from unfamiliar photos. B) A 0.93 score says little about accuracy on photos unlike the calibration set; stratified sampling of auto-approved estimates must measure it. C) Asking the model whether it is sure about the vehicle type, and approving only when it says yes, confirms that the score is trustworthy. D) The estimates can be approved, because the threshold was calibrated against labelled data and 0.93 clears it by a comfortable margin.

Q327. At a regional hospital, coders currently check every billing code an assistant proposes for outpatient, surgical and emergency encounters; its headline figure is 96% overall accuracy. Finance proposes letting codes it is highly confident about post unchecked. Compliance will sign off only with the right protections. Which two belong in that sign-off? (Select TWO.) A) Have the assistant state in words how certain it is about each code, and auto-accept only the codes it describes as certain. B) Measure accuracy against a labelled set separately for each encounter type and code field, and set confidence thresholds per field from that data. C) Keep a stratified random sample of automatically accepted codes under ongoing human review to track error rates by segment. D) Require the 96% overall accuracy figure to stay steady for two consecutive quarters before any reduction in review takes effect.

Q328. A procurement agent extracts unit prices from supplier quotes and writes lines such as "Quote Q-118, page 3, dated 2 May, lists the unit price as $4.10." The ERP system that imports this output keeps the price but never keeps the quote reference, so buyers cannot check where a price came from. What is the root cause? A) The agent should maintain a separate lookup table from each price value to its quote so buyers can search it whenever they need to. B) The source reference exists only inside a sentence, and the consumer parses values, so the fix is emitting each price with separate source fields. C) The ERP import mapping is misconfigured, and adding a free-text notes column to carry each line through unchanged would solve it. D) The citations should be moved into numbered footnotes at the end of the output so they are clearly set apart from the price values.

{
  "item": "M6 hex bolt",
  "unit_price": 4.1,
  "currency": "USD",
  "source": {
    "quote_id": "Q-118",
    "page": 3,
    "quote_date": "2025-05-02"
  }
}

Q329. Before deleting a feature flag, a developer needs every place in a large monorepo that reads NEW_CHECKOUT_ENABLED, including config files, tests and docs. The search will take dozens of grep calls and file reads. Which statement about handing this search to an exploration subagent is accurate? A) The subagent may read files that the main session's permission settings block, which makes the search more complete. B) The subagent always uses fewer total tokens than the same search in the main session, which is the main reason to delegate. C) The subagent gets a much larger context window than the main session, so it can hold the entire monorepo at once. D) The subagent's greps and file reads fill its own context, and only its concise list of findings enters the main session.

Q330. Early in a long Claude Code session building a payments reconciliation job, the developer and the agent agreed that money amounts must always be integer minor units, never floating point. After several compactions, the agent writes total = float(row.amount). Which practice would most reliably have prevented this? A) Restate the rule once more in the chat halfway through the session so it is more recent when the next compaction happens. B) Compact more often, so that each summary is shorter and closer to the moment when the money-handling rule was agreed. C) Record the rule in the project's CLAUDE.md or a decisions file that is reloaded every session, not only in the chat history. D) Lower the temperature for the coding steps so the agent follows the patterns it established earlier in the session more closely.

Q331. An incident agent first reads 25 log files and metrics exports to reconstruct an outage, then writes the postmortem. By the writing phase its context is full of raw log lines, and the draft gets several timestamps and the order of events wrong. How should the workflow be structured? A) Move to a model tier with a larger context window so the raw logs and the postmortem draft fit together without crowding. B) End the investigation with a structured brief of timeline, evidence and file references, and write the postmortem from that brief. C) Interleave the phases so each paragraph of the postmortem is written right after the log file it relies on has been read. D) Have the writer reread all 25 files at the start of the writing phase so the timeline details are fresh in its context.

Q332. An agent has spent two hours investigating a legacy insurance-claims system across many tool calls. Which observation is the strongest sign of context degradation rather than some other problem? A) A reply stops in the middle of a sentence and the response shows a stop reason indicating the output token limit was reached. B) Early answers named exact files and classes such as ClaimAdjuster; recent answers on the same area say it "likely follows common patterns." C) Several consecutive tool calls fail with rate-limit responses, and the agent waits a while before trying the same calls again. D) The agent declines to print the database password stored in the configuration file and suggests using an environment variable instead.

Q333. A due-diligence agent assembles an acquisition report from three sources: court-records search, financial filings and press coverage. On one run the court-records service was down, and the report went to the client looking complete. Which output design is right? A) Tag each section with which sources contributed and flag that litigation coverage is missing and related conclusions are weaker. B) Have the agent fill the litigation section from its general knowledge of the target company, labelled as a model estimate. C) Hold the report entirely until all three sources are available again, returning only an error message to the client until then. D) Put a standard notice on every report saying that sources may be incomplete, so clients always read results with some caution.

Q334. A news-briefing pipeline has three collector agents that each return articles with URLs. To save tokens, a condenser step turns each collector's output into short bullet highlights before an editor agent writes the daily brief. The brief never contains links, even though every collector output did. Which change fixes this at the right point? A) Have the condenser output each highlight as a record paired with its source URLs, and have the editor keep those pairs in the brief. B) Add an instruction to the editor's system prompt saying that every statement in the brief must include a link to its source. C) After the brief is written, run a step that searches the web for each sentence and attaches the best-matching article link to it. D) Give the editor a larger context window so it can take in all collector outputs directly and cite from them in full.

Q335. A contract-review pipeline sends whole 120-page service agreements in one request. Clauses near the start and end are extracted reliably, but indemnification terms, which usually sit around pages 55–70, are missed in most contracts. Token counts confirm nothing is truncated. Which change targets the actual cause? A) Increase max_tokens on the request so the model has more room to write out the indemnification clauses it finds. B) Repeat the instruction to extract indemnification terms twice in the system prompt so the model gives it more importance. C) Split each agreement by section and run a focused extraction pass over the middle sections, placing the question after that text. D) Move to a model with a larger context window so each 120-page agreement uses a smaller fraction of the space available.

Q336. A support platform auto-applies ticket categories when model confidence exceeds 0.85; a single labelled validation run showed 96% accuracy. The company launches new product lines every quarter, and operations wants to catch new mistakes in the auto-applied categories early. Which monitoring plan does that best? A) Rerun the original validation set every month and raise an alert if its measured accuracy ever drops below 96%. B) Count how often support agents manually recategorise tickets and investigate whenever that number rises noticeably. C) Each week, have people label a random sample of auto-applied categories drawn from every product line, and track error rates per line. D) Track the average confidence of auto-applied categories each week, and alert when the average falls below its level at launch.

Q337. A fleet-maintenance agent checks 300 vehicles in one run. Each get_vehicle_status call returns about 60 sensor fields, but the task only uses battery_pct, odometer_km and fault_codes. Accuracy drops noticeably for vehicles late in the run. Where should the fix go? A) In the session settings, compacting the conversation after every 50 vehicles so the accumulated tool output is summarised. B) In the request parameters, raising max_tokens so the model has more space to reason about each vehicle's full status. C) In the system prompt, with a line telling the model to ignore every sensor field except the three that matter for maintenance. D) In the tool-result handling, reducing each response to the three needed fields before it is added to the conversation.

KEEP = ('battery_pct', 'odometer_km', 'fault_codes')

def trim_status(raw: dict) -> dict:
    return {k: raw.get(k) for k in KEEP}

# in the loop, append trim_status(result) as the tool_result content, not result

import json
import os
from typing import Annotated

import anthropic
from anthropic import beta_tool
from pydantic import Field

from app.fleet import telematics  # your own client for the vehicle API

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY
MODEL = os.environ.get("CLAUDE_MODEL", "claude-sonnet-5")
KEEP = ("battery_pct", "odometer_km", "fault_codes")


@beta_tool(
    name="get_vehicle_status",
    description="Read one vehicle's battery level, odometer and active fault codes.",
)
def get_vehicle_status(vehicle_id: Annotated[str, Field(description="The fleet vehicle id, e.g. EV-0412.")]) -> str:
    raw = telematics.status(vehicle_id)  # about sixty fields
    return json.dumps({k: raw.get(k) for k in KEEP})


runner = client.beta.messages.tool_runner(
    model=MODEL, max_tokens=2048, tools=[get_vehicle_status],
    messages=[{"role": "user", "content": "Which vehicles need service today?"}],
    max_iterations=50,
)
final = runner.until_done()

Q338. A mortgage-application extractor gets applicant_name and loan_amount right almost every time, but employment_start_date and self-employed income are often wrong. Underwriters can review roughly 15% of extracted fields. Which routing design spends that capacity where the errors are? A) Emit a confidence score for each field, set thresholds per field against a labelled set, and review only fields below them. B) Send every application that includes self-employed income to review in full, so underwriters see all fields with context. C) At the end of each application, ask the model which applications it feels unsure about overall and send those to underwriters. D) Pick a random 15% of all applications each day and have underwriters check every field in each application picked.

{
  "applicant_name": {
    "threshold": 0.6,
    "measured_accuracy_above": 0.995
  },
  "loan_amount": {
    "threshold": 0.7,
    "measured_accuracy_above": 0.99
  },
  "employment_start_date": {
    "threshold": 0.92,
    "measured_accuracy_above": 0.97
  },
  "self_employed_income": {
    "threshold": 0.95,
    "measured_accuracy_above": 0.96
  }
}

Q339. A warehouse agent reads delivery paperwork. Printed domestic packing slips are 70% of volume at 98.8% accuracy, customs forms are 20% at 96.9%, and handwritten slips are 10% at 61%. Overall accuracy is about 94.6%. The plan is to auto-post every extraction with confidence of at least 0.9. Which fact makes the plan unsafe? A) Inventory reconciliation will not catch errors quickly enough, so validation checks after posting must be added first. B) A single global threshold would let handwritten slips through automatically, even though roughly two in five of them are wrong. C) The 0.9 threshold is too low for all three paperwork types, so it should simply be raised to 0.95 for every type. D) Handwritten slips are only 10% of volume, so their errors are too small a share of the total to matter for the plan.

Q137. A support agent has lookup_order, process_refund, and escalate_to_human. The customer’s first message is: "I need to speak to a human about my refund." What should the agent do first? A) Call lookup_order first so the human receives the order details with the handoff B) Call escalate_to_human immediately, passing what it already knows C) Explain that it can process refunds itself and ask to try before transferring D) Ask why the customer prefers a human, then decide whether escalation is needed

Q138. A customer writes: "This is the THIRD time my package is late. This is ridiculous." They have not asked for a person. Checking order status and offering a reship are within the agent’s tools and policy. What should the agent do? A) Escalate immediately, because strong negative sentiment signals a complex case B) Run a sentiment score and escalate if it falls below the configured threshold C) Skip the emotion and reply with the tracking number only D) Acknowledge the frustration, look up the order, and offer a concrete fix; escalate if the customer then asks for a human

Q139. Continuing the late-package case: the agent acknowledged the frustration and offered a reship. The customer replies: "No. I just want a person." What now? A) Escalate now; the customer has stated a preference for a human after the offer B) Improve the offer with a discount code and try once more C) Process the reship anyway, since it is within policy, and then escalate D) Ask which department the customer wants before escalating

Q140. To improve handoff quality, a team adds to the system prompt: "Before escalating, always make at least one tool call to gather context." What is the problem with this instruction? A) There is none; handoffs will now always include order data B) There is none; the model will ignore it for explicit requests anyway C) It delays explicit requests for a human; exempt them so the agent escalates at once with what it knows D) It is fine as long as the tool calls are fast

Q141. You are writing the escalation section of a support agent’s system prompt. Which instruction captures both sides of the honor-versus-resolve rule? A) "Escalate whenever the customer seems upset; otherwise resolve the issue yourself." B) "Always attempt a resolution first; escalate only if your tools fail." C) "Escalate any conversation that involves a refund; resolve everything else." D) "If the customer asks for a human, escalate immediately without investigating. If they are frustrated but the issue is within your tools and policy, acknowledge it and offer a resolution; escalate if they repeat the request."

Q142. Midway through a return, the agent has found the order but has not yet called create_return. The customer says: "Actually, just get me a human." What should happen? A) Finish the return first so the work is not lost, then escalate B) Stop, escalate now, and record in the handoff that the order was found and no return was submitted C) Escalate with no summary, since the customer abandoned the flow D) Ask the customer to confirm they want to abandon the return

Q143. A customer asked for a human in their first message, so the agent escalated without investigating. Human agents cannot see the chat transcript. What belongs in the escalate_to_human call? A) The customer ID if known, the issue in the customer’s words, a reason of explicit request, and a note that no investigation or action was taken B) A link to the raw transcript and nothing else C) Nothing; the human will ask the customer to start over D) The agent’s guess at the root cause, produced by running diagnostics before the call

{
  "customer_id": "C-20931",
  "reason": "customer_request",
  "customer_statement": "I need to speak to a human about my refund.",
  "steps_taken": [],
  "actions_executed": [],
  "recommended_action": "Review refund request for order the customer will reference."
}

Q144. A customer asks the agent to match a competitor’s lower price. Company policy covers price adjustments only when the company’s own site price drops, and says nothing about competitors. What should the agent do? A) Deny the request, because the policy covers only own-site price drops B) Approve it, because the policy’s intent is price fairness C) Escalate: the policy is silent on competitor matching, which is a policy gap D) Ask for a screenshot of the competitor price and approve if it is valid

Q145. An agent has tried three different ways to locate a customer’s account: by email, by phone, and by order number. Each returned no match, and the customer is patient and polite. What should happen next? A) Escalate with a handoff listing the three lookups and their results, because the agent cannot make meaningful progress B) Keep retrying the same lookups with exponential backoff until one succeeds C) Tell the customer the account does not exist and end the conversation D) Wait to escalate until the customer becomes frustrated

Q146. A team proposes routing every conversation to a human whenever a sentiment model scores the customer’s message below −0.6. What is the main weakness of this design? A) Sentiment models are too slow for real-time chat B) Sentiment is an unreliable proxy for case complexity: angry customers often have simple issues, and calm ones can hit policy gaps C) Sentiment scoring violates customer privacy regulations D) The design is sound, but the threshold should be −0.8

Q147. An agent rates its own confidence from 1–10 on each case and escalates below 6. In review, it escalates many easy cases and confidently handles several hard ones incorrectly. What explains this? A) The threshold of 6 is too high and should be lowered B) Temperature is too high, which makes the confidence scores noisy C) Confidence should be computed in a separate batch job D) An LLM’s self-reported confidence is poorly calibrated: it does not track how complex or risky the case actually is

Q148. For the miscalibrated agent in the previous question, which change is the most effective first step to improve when it escalates? A) Train a separate classifier on historical tickets to pre-route every conversation B) Add a sentiment model and escalate when the customer is frustrated C) Add explicit escalation criteria to the system prompt, with few-shot examples showing when to escalate and when to resolve D) Lower the self-rated confidence threshold from 6 to 4

Escalate to a human when:
- the customer explicitly asks for a person (do this immediately, before any lookup);
- policy is silent or ambiguous on the specific request;
- you cannot make meaningful progress after reasonable attempts.
Do NOT escalate only because the customer is upset or because the case has many steps.

Example 1: "Get me a human." -> escalate now, reason=customer_request, steps_taken=[].
Example 2: "Third late package, this is ridiculous." -> acknowledge, check the order, offer a reship. Escalate if they then ask for a person.
Example 3: "Will you match Competitor X's price?" (policy covers own-site drops only) -> escalate, reason=policy_gap.

Q149. get_customer(name="Maria Garcia") returns three matching accounts. The customer wants to change a shipping address. What should the agent do? A) Select the most recently active account and proceed B) Ask the customer for an additional identifier, such as email or an order number, before acting C) Escalate to a human, because the lookup is ambiguous D) Apply the address change to all three accounts

Q150. Which of these situations is a valid trigger for escalating to a human? A) The customer asks for a refund exception that falls outside the documented return window B) The customer writes in all capital letters C) Resolving the case will take more than five tool calls D) The model’s self-reported confidence is below 0.7

Q151. Policy says refunds over $500 require a human. The system prompt says "Never issue refunds over $500," yet about 1 in 200 conversations still issues one. What is the right fix? A) Rewrite the rule in capital letters and repeat it at the end of the prompt B) Add few-shot examples of the agent refusing large refunds C) Lower the temperature to make the agent follow the rule consistently D) Add a hook or harness gate that blocks process_refund above $500 and redirects to escalate_to_human

REFUND_LIMIT = 500

def pre_tool_use(tool_name, tool_input, case):
    if tool_name == "process_refund" and tool_input["amount"] > REFUND_LIMIT:
        return {
            "decision": "block",
            "redirect": "escalate_to_human",
            "handoff": {
                "customer_id": case["customer_id"],
                "reason": "threshold_exceeded",
                "proposed_refund": tool_input["amount"],
                "steps_taken": case["steps_taken"],
            },
        }
    return {"decision": "allow"}

Q152. Human agents say escalations are useless: each arrives as "Customer needs help with refund." They cannot see the chat transcript. What should each escalation carry? A) A link to the raw conversation log B) The agent’s final apology message to the customer C) A structured summary: customer ID, issue, root cause found, actions taken and their results, amounts, recommended next action, and the escalation reason D) The customer’s sentiment score across the conversation

Q153. Sometimes a support session ends with a tool error or stop_reason: "max_tokens" mid-case. The customer is left with neither a resolution nor an escalation. What design fixes this? A) Raise max_tokens so sessions are less likely to be cut off B) An orchestration-level safeguard: if the loop ends without a resolved status, the harness creates an escalation with the handoff packet C) Add a prompt instruction: "Always end with a resolution." D) Restart the whole conversation from the beginning

Q154. Where should a support agent’s escalation criteria and their examples live so they shape every decision? A) In the system prompt: explicit criteria plus two to four few-shot examples covering both sides of the boundary, with the reasoning for each B) Only in the escalate_to_human tool description, as "Use when needed" C) Appended to each user message by the frontend D) In a fine-tuned model trained on past escalations

Q155. The escalate_to_human tool needs a reason field that analytics can group, while still representing cases nobody anticipated. Which schema design fits? A) A free-text reason string B) A required enum of exactly three values: customer_request, policy_gap, no_progress C) A boolean is_urgent D) An enum of the known triggers plus "other", with a nullable reason_detail string

{
  "name": "escalate_to_human",
  "input_schema": {
    "type": "object",
    "properties": {
      "customer_id": {
        "type": [
          "string",
          "null"
        ]
      },
      "reason": {
        "type": "string",
        "enum": [
          "customer_request",
          "policy_gap",
          "no_progress",
          "threshold_exceeded",
          "other"
        ]
      },
      "reason_detail": {
        "type": [
          "string",
          "null"
        ]
      },
      "steps_taken": {
        "type": "array",
        "items": {
          "type": "string"
        }
      },
      "recommended_action": {
        "type": "string"
      }
    },
    "required": [
      "reason",
      "steps_taken",
      "recommended_action"
    ]
  }
}

Q156. process_refund must never run until get_customer has returned a verified customer ID in the current session. Which design guarantees this? A) A system-prompt instruction: "Always verify the customer before refunding." B) Few-shot examples showing verification happening before every refund C) A programmatic prerequisite in the harness that blocks process_refund until session state holds a verified customer ID D) Asking the model to confirm verification in its reasoning before calling the tool