Pro Domain 4 · Evaluation, Testing & Optimization (16%)

This domain asks an architect to do three things. First, prove that a Claude system works. Second, keep proving it after every change. Third, make the system cheaper and faster without breaking that proof. The Professional profile expects you to define the numbers (accuracy, latency, cost, safety, security). You build the dataset and the framework that produce those numbers. You run controlled experiments. You trace a live failure back to its real cause. You read the logs, so that a 2 AM incident becomes a query instead of a guess. This page builds on Foundations Domain 5 · reliability engineering and the prompt-iteration loop. Foundations taught that you measure. This page teaches what you measure, how each number is computed, and which artifact (file, log or record) holds it.

Define evaluation metrics — accuracy, latency, cost, safety, security

Define evaluation metrics (accuracy, latency, cost, safety, security)

Who defines each metric, and where it lives. The architect writes the metric definitions in the evaluation spec. The spec is a versioned document in the repo, next to the golden set. The scoring code in the evaluation harness implements those definitions. The raw inputs come from three places: the usage block in the model response (tokens, cache reads), the harness clock (latency), and the scoring functions (accuracy, safety, security). The Messages API never returns “accuracy”. Your code computes it by comparing the answer to a labeled expected answer.

One record per case, five families in one shape. The harness writes one JSON record for each evaluated case. Everything that comes later (dashboards, A/B comparisons, regression alerts) reads this same shape. Here is an illustrative (made-up) record for one Northwind case:

{
  "eval_run": "claims-triage-2026-09-15-prompt-v7",
  "case_id": "gs-0412",
  "category": "policy-exception",
  "expected": {
    "triage": "escalate",
    "policy_ref": "POL-217 s4.2"
  },
  "observed": {
    "triage": "escalate",
    "policy_ref": "POL-217 s4.2"
  },
  "scores": {
    "accuracy": 1.0,
    "faithfulness": 0.9,
    "safety_pass": true,
    "security_pass": true
  },
  "latency_ms": {
    "first_token": 410,
    "total": 2380
  },
  "usage": {
    "input_tokens": 6120,
    "cache_read_input_tokens": 5800,
    "output_tokens": 210
  },
  "cost_units": 0.0112,
  "model_tier": "larger",
  "prompt_version": "v7",
  "index_version": "idx-2026-09-14",
  "stop_reason": "end_turn"
}

Your code produces scores. Your code copies usage and stop_reason from the response, so the record can later explain a cost or a cut-off answer. index_version and prompt_version let you trace a regression back to what changed. Your code computes cost_units from usage and your contract rate. The API never returns money amounts.

Decision table — which metric answers which question:

The wrong choice and its failure. A team reports “94% accurate” from a spreadsheet of thumbs-up votes. That number only stands in for satisfaction. It comes from a sample of users who chose to vote. It cannot show that policy-exception cases are only 61% correct, because those cases are rare. The exam may offer “increase the sample size” against “define a labeled set with a per-case scoring rule and per-category reporting.” The second option wins. More of a bad measurement is still a bad measurement. See the Foundations note on why an overall score can hide a subgroup in confidence calibration and stratified sampling. Card: evaluation-metrics.

Design evaluation datasets and test frameworks using mixed methodologies

Design evaluation datasets and test frameworks using mixed methodologies

Who builds it and where it lives. The architect defines the slice plan. Domain reviewers supply the labels. The harness owner versions the file. The golden set is a versioned data file in the repo (JSONL is typical). The rubric for the judge is a prompt file. The human-review sample is a queue in your review tool. What goes into the set is a design decision. It should not depend on what was easy to collect:

Mixed methods — four graders, each with a job. Read the eval and golden set cards for the Foundations words. This table shows how the graders split the work:

The judge request is a normal Messages request. The rubric goes in system. The candidate answer and the reference go in the user turn, which your harness builds. The request forces the verdict into a schema, so the harness can parse it without regex. Illustrative request:

{
  "model": "YOUR_SUPPORTED_MODEL",
  "max_tokens": 400,
  "system": "You grade customer-reply drafts for an insurance claims assistant. Score faithfulness (every factual statement is supported by the supplied policy excerpt), completeness (the customer knows the decision and the next step), and tone (professional, no promises beyond the excerpt). Score each 0 to 1. Cite the sentence that lowers any score. Do not reward length.",
  "messages": [
    {
      "role": "user",
      "content": "POLICY EXCERPT:\nPOL-217 s4.2: Water damage from an appliance failure is covered when the appliance was maintained per manufacturer guidance. Claims above 5000 require an adjuster visit.\n\nEXPECTED TRIAGE: escalate (amount 7200)\n\nCANDIDATE REPLY:\nThank you for your claim. Appliance water damage is covered under your policy. Because the amount exceeds 5000, an adjuster will contact you within two business days to schedule a visit. Please keep the failed appliance available for inspection.\n\nReturn the verdict JSON."
    }
  ],
  "output_config": {
    "format": {
      "type": "json_schema",
      "schema": {
        "type": "object",
        "properties": {
          "faithfulness": {"type": "number"},
          "completeness": {"type": "number"},
          "tone": {"type": "number"},
          "evidence": {"type": "string"}
        },
        "required": ["faithfulness", "completeness", "tone", "evidence"],
        "additionalProperties": false
      }
    }
  }
}

The candidate reply promised “within two business days”. The excerpt does not say that. A good judge lowers faithfulness and quotes that sentence in evidence. Notice what the judge cannot do. It cannot tell you whether the retrieved excerpt was the right excerpt. That is a retrieval metric. Your harness scores it separately, by comparing the retrieved chunk ids to the expected policy_ref.

The wrong choice and its failure. A team replaces human labels with judge labels to save money. This removes the only grader that can calibrate (check and correct) the judge. Suppose the judge and the model share a blind spot: both think “two business days” is fine. Then the metric reads 100%, while customers get promises the policy never made. Taking a bigger sample from the same easy source is the same mistake, only larger. Cards: evaluation-dataset · mixed-method-evaluation. Foundations background: evaluation of agents.

Conduct A/B testing and iterative improvements

Conduct A/B testing and iterative improvements

Who decides, and where the decision lives. The architect writes the experiment record before the run. The harness fills in the results. The decision rule is code in the CI gate, not a meeting. Offline A/B (both versions run over the golden set) comes first, because it is free and reproducible. Online A/B (a traffic split) comes second, because only live traffic contains the inputs you did not imagine. Foundations covered the shape of the loop in prompt iteration as engineering. The Professional level adds the record and the rule.

The experiment record — one file per experiment, versioned together with the prompt it tests. Illustrative:

experiment: claims-triage-prompt-v7
hypothesis: Adding two policy-exception examples raises exception accuracy without hurting routine accuracy.
baseline: prompt-v6
candidate: prompt-v7
change: one change only - two few-shot examples added to the system prompt
dataset: golden-set-2026-09-01 (held-out slice reported separately)
primary_metric: accuracy.policy-exception
decision_rule:
  ship_if:
    - primary_metric gain >= 0.02
    - no category accuracy drop > 0.01
    - safety_pass == 1.0 on adversarial slice
    - security_pass == 1.0 on adversarial slice
    - p95_latency_ms increase <= 10 percent
    - cost_per_case increase <= 5 percent
  canary: 10 percent of traffic for 48 hours, same rule on live metrics
results:
  offline:
    accuracy.overall: {baseline: 0.93, candidate: 0.96}
    accuracy.policy-exception: {baseline: 0.71, candidate: 0.67}
    p95_latency_ms: {baseline: 2410, candidate: 2530}
    cost_per_case: {baseline: 0.0108, candidate: 0.0119}
  decision: do-not-ship
  reason: primary metric regressed despite overall gain; investigate example selection

Read the record the way the exam does. Overall accuracy went up. The primary metric went down. A team that ships based on “overall” has given up the rare, expensive case to improve the common, cheap one. The decision rule caught this. The rule named the slice that matters, and it set a guard for every other slice.

The wrong choice and its failure. “Ship because the overall number went up” and “ship after the product manager approves” both skip the rule. “Rerun the judge with temperature zero until it agrees” is not iteration. It is re-grading the grader until it says yes. Card: ab-testing. Also see the Foundations evaluation-workflow card for the basic words.

Diagnose system issues — prompt failure, hallucination, model mismatch, retrieval failure

Diagnose system issues (prompt failure, hallucinations, model mismatch)

Who diagnoses, and with what. The on-call engineer reads the observability record for each call (next section). It holds the retrieved chunk ids, the exact prompt version, the tier, the stop_reason, and the usage. The architect owns the decision table below. A diagnosis is not complete until the golden set has a new regression case that would have caught the failure.

The diagnostic ladder as a checklist. The order matters: each step is cheaper, and more likely to find the cause, than the next one:

1. What changed?            deploy log: prompt version, tier, index version, tool list, SDK version
2. Which inputs fail?       eval breakdown by category; are the failures new inputs or old ones?
3. What did the model see?  the exact request: system, messages, retrieved chunks, tools
4. What did it return?      content blocks, stop_reason, usage
5. Which component owns it? retrieval -> prompt -> tier -> control gate
6. Prove it                 add a regression case; the fix must turn it green

Retrieval failure needs special attention. With RAG, the model can only be as correct as its context. Sometimes retrieval returns old or unrelated chunks. Then the model does exactly its job: it reasons faithfully over the wrong material. The output looks like hallucination, but the model is really following its input. Foundations covered the pipeline in millions of PDFs into RAG. The point for diagnosis: the fix lives in the index, the chunking, or the embedding-model version. It never lives in the prompt.

{
  "alert": "faithfulness_live_sample_below_floor",
  "service": "claims-triage",
  "window": "2026-09-14T06:00Z/2026-09-14T09:00Z",
  "observed": 0.71,
  "floor": 0.92,
  "unchanged": ["prompt_version=v6", "model_tier=larger", "p95_latency_ms=2390"],
  "changed_since_last_green": ["index_version idx-2026-09-07 -> idx-2026-09-14"]
}

{
  "trace_id": "tr-8f31",
  "query": "washing machine hose burst, damage 7200",
  "retrieval": {
    "index_version": "idx-2026-09-14",
    "embedding_model_version": "emb-v2",
    "query_embedding_model_version": "emb-v1",
    "top_k": [
      {"chunk_id": "POL-217:s4.2:c3", "score": 0.61},
      {"chunk_id": "POL-217:s4.2:c9", "score": 0.58},
      {"chunk_id": "POL-140:s1.1:c0", "score": 0.57}
    ],
    "expected_chunk_present": false
  }
}

The wrong choice and its failure. Adding “do not make things up” to the system prompt treats a data problem as if the model were disobeying. The model was already following instructions. Moving to a larger tier costs more, and it reasons just as faithfully over the same wrong chunks. Lowering temperature only makes the wrong answer more consistent. Cards: failure-diagnosis · hallucination-vs-retrieval · model-mismatch.

The refresh archetype — “changed X, unchanged Y” tells you where to look

Diagnose system issues (prompt failure, hallucinations, model mismatch) — the exam’s favorite framing

Read the stem as a table. Each phrase in the stem either points at a component or rules one out:

The words the answer key uses to explain. The official explanation for a retrieval-first answer says the other options “would not be triggered specifically by” the change described. Use that phrase as a test. For each option, ask: “does a document refresh trigger this?” If not, it is a wrong option, even if it sounds reasonable on its own.

What a retrieval-side check really is. Your logs must carry two numbers, so that the first investigation step is a query, not a guess:

def retrieval_health(trace, expected_chunk_id):
    hit = any(c["chunk_id"] == expected_chunk_id for c in trace["retrieval"]["top_k"])
    parity = (
        trace["retrieval"]["embedding_model_version"]
        == trace["retrieval"]["query_embedding_model_version"]
    )
    top_score = max((c["score"] for c in trace["retrieval"]["top_k"]), default=0.0)
    return {"expected_chunk_present": hit, "embedding_parity": parity, "top_score": top_score}

expected_chunk_present is the retrieval accuracy metric. embedding_parity is the preventive check that would have stopped the Monday failure completely. When top_score keeps falling after a refresh, that is the early-warning signal.

The wrong choice and its failure. In a refresh stem, every option that is not about retrieval fails in the same way. It changes a component that did not change. So the symptom stays, and the team has spent a sprint on the wrong fix. Card: hallucination-vs-retrieval.

Optimize token usage, latency, and cost-performance trade-offs

Optimize token usage, latency, and cost-performance trade-offs

Who chooses the lever, and where it is set. The architect chooses. The setting lives in one specific place: a request field, a routing rule in the harness, or a job scheduler. Foundations explained that tokens and money are different units, in tokens versus money and the budget mindset. Here is the list of levers with their trade-offs:

Read the cost lever from the response. The only cost signal you can trust is the usage block on each response. Illustrative code that adds up the numbers for one route:

def route_cost_report(records, rate_in, rate_cached, rate_out):
    total_in = sum(r["usage"]["input_tokens"] for r in records)
    cached = sum(r["usage"].get("cache_read_input_tokens", 0) for r in records)
    out = sum(r["usage"]["output_tokens"] for r in records)
    cost = (total_in - cached) * rate_in + cached * rate_cached + out * rate_out
    resolved = sum(1 for r in records if r["scores"]["accuracy"] >= 1.0)
    return {
        "calls": len(records),
        "cache_hit_ratio": round(cached / total_in, 3) if total_in else 0.0,
        "cost_units": round(cost, 4),
        "cost_per_resolved": round(cost / resolved, 4) if resolved else None,
        "truncation_rate": round(
            sum(1 for r in records if r["stop_reason"] == "max_tokens") / len(records), 3
        ),
    }

Your code supplies the rates from your contract. They are never fixed values in the KB. Three computed numbers catch regressions. If cache_hit_ratio falls sharply, someone put changing content before the cache marker. If truncation_rate rises, max_tokens is too low. If cost_per_resolved rises while total cost stays flat, accuracy fell.

A token budget per route is a design artifact. The architect writes it, and the harness enforces it:

{
  "route": "routine-claim",
  "model_tier": "faster",
  "budget": {
    "system_prefix_tokens": 5400,
    "cached_prefix_expected": true,
    "retrieved_context_tokens_max": 3000,
    "conversation_tokens_max": 4000,
    "max_tokens": 600
  },
  "guards": {
    "accuracy_floor": 0.95,
    "p95_latency_ms_max": 2500,
    "truncation_rate_max": 0.0
  }
}

The wrong choice and its failure. A team moves all traffic to the faster tier because cost fell 60%. That accepts the policy-exception regression that the per-category eval would have shown. Cutting the policy document short to save tokens deletes the clause the answer needed. Lowering max_tokens without watching stop_reason turns valid answers into cut-off JSON. In each case, someone pulled a lever without its guard metric. Card: cost-performance-tuning. Foundations mechanics: Message Batches.

Monitor system performance using logging and observability tools

Monitor system performance using logging and observability tools

Who emits, who reads, where it lives. The harness writes one structured event for each model call and each tool call. The observability platform stores and indexes these events. It is whatever your organization runs: a log store, a metrics system, a tracing backend. The on-call engineer and the eval harness read the events. Foundations set the standard in observability — every action explainable, auditable, replayable. The Professional level adds the field list and the alert rules.

The event — illustrative, one model call inside one run:

{
  "event": "model_call",
  "trace_id": "tr-8f31",
  "run_id": "run-2026-09-14-001882",
  "span": "triage-decision",
  "timestamp": "2026-09-14T08:12:44Z",
  "service": "claims-triage",
  "route": "policy-exception",
  "prompt_version": "v6",
  "model_tier": "larger",
  "index_version": "idx-2026-09-14",
  "tools_offered": ["lookup_policy", "create_case"],
  "retrieval": {
    "top_k_chunk_ids": ["POL-217:s4.2:c3", "POL-217:s4.2:c9", "POL-140:s1.1:c0"],
    "top_score": 0.61
  },
  "request_summary": {
    "system_hash": "sha256:1c9e",
    "message_count": 3,
    "input_hash": "sha256:a77d"
  },
  "response": {
    "stop_reason": "tool_use",
    "tool_requested": "create_case",
    "usage": {
      "input_tokens": 6120,
      "cache_read_input_tokens": 5800,
      "output_tokens": 210
    }
  },
  "gate": {
    "decision": "allowed",
    "rule": "create_case requires triage != approve or amount < 1000"
  },
  "latency_ms": {
    "first_token": 410,
    "total": 2380
  },
  "pii_redacted": true
}

Why each field is there. Each one answers a diagnostic question from the ladder above:

Alert rules. These five catch this domain’s failures early. Each rule names the metric, the comparison, and the first action. The numbers are Northwind’s; yours will be different:

alerts:
  - name: cache_hit_ratio_drop
    metric: cache_read_input_tokens / input_tokens, per route, 15 min
    condition: below 0.6 when the previous day averaged above 0.85
    first_action: diff the system prompt and tool list for volatile content before the cache marker
  - name: truncation_rate
    metric: share of calls with stop_reason == max_tokens
    condition: above 0 for structured-output routes
    first_action: raise max_tokens on the route; check for runaway output
  - name: faithfulness_live_sample
    metric: judge score on a 5 percent stratified live sample
    condition: below 0.92 over 3 hours
    first_action: compare retrieval top_score and index_version to last green window
  - name: gate_denials_spike
    metric: gate.decision == denied, per tool, per hour
    condition: 3x the 7 day baseline
    first_action: read the denied trajectories; suspect injection or a prompt regression
  - name: p95_latency
    metric: latency_ms.total p95, per route
    condition: above the route SLA for 10 minutes
    first_action: check tier, retrieved_context_tokens, and provider status

The wrong choice and its failure. A monthly spend alert catches a caching regression thirty days late. An average-latency dashboard hides the p95 tail (the slowest calls), and the SLA is written against that tail. Logging raw customer messages helps debugging, but it creates a data-protection incident. Log hashes and spans with personal data removed instead. Alerting on the judge score alone treats evidence as a control. The gate log is the evidence that the control worked. Card: observability-tooling. Where these events flow in a deployment: the support agent on GCP.

The evaluation harness — the whole loop in one file

Design evaluation datasets and test frameworks using mixed methodologies — the framework half

Who owns it and where it runs. The platform team owns the harness code. The architect owns the decision rule inside it. It runs in CI, and also on a schedule against production samples. The judge call is a normal Messages request. The harness calls the system under test through the same entry point that production uses, so the eval measures the real system. Illustrative sketch; run_system and call_judge wrap your application and your judge request:

import json
import sys
import time
from collections import defaultdict


def load_cases(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]


def score_case(case, result, call_judge):
    scores = {}
    if "expected_triage" in case:
        scores["accuracy"] = 1.0 if result["triage"] == case["expected_triage"] else 0.0
    if "expected_chunk_id" in case:
        ids = [c["chunk_id"] for c in result["retrieval"]["top_k"]]
        scores["retrieval_hit"] = 1.0 if case["expected_chunk_id"] in ids else 0.0
    if case.get("grader") == "judge":
        verdict = call_judge(case, result["reply_text"])
        scores["faithfulness"] = verdict["faithfulness"]
    if case.get("slice") == "adversarial":
        scores["security_pass"] = result["gate"]["decision"] == "denied"
    return scores


def aggregate(records):
    by_cat = defaultdict(lambda: defaultdict(list))
    for r in records:
        for name, value in r["scores"].items():
            by_cat[r["category"]][name].append(float(value))
    return {
        cat: {name: sum(v) / len(v) for name, v in metrics.items()}
        for cat, metrics in by_cat.items()
    }


def decide(summary, baseline, rule):
    failures = []
    for cat, metrics in summary.items():
        for name, value in metrics.items():
            floor = baseline.get(cat, {}).get(name)
            if floor is not None and value < floor - rule["max_drop"]:
                failures.append(f"{cat}.{name}: {value:.3f} < baseline {floor:.3f}")
    for cat in rule["must_be_perfect"]:
        for name in ("security_pass", "safety_pass"):
            value = summary.get(cat, {}).get(name)
            if value is not None and value < 1.0:
                failures.append(f"{cat}.{name}: {value:.3f} != 1.0")
    return failures


def main(cases_path, baseline_path, run_system, call_judge, rule):
    cases = load_cases(cases_path)
    with open(baseline_path) as f:
        baseline = json.load(f)
    records = []
    for case in cases:
        started = time.monotonic()
        result = run_system(case["input"])
        elapsed_ms = int((time.monotonic() - started) * 1000)
        records.append({
            "case_id": case["id"],
            "category": case["category"],
            "scores": score_case(case, result, call_judge),
            "latency_ms": elapsed_ms,
            "usage": result["usage"],
            "stop_reason": result["stop_reason"],
        })
    summary = aggregate(records)
    failures = decide(summary, baseline, rule)
    with open("eval-records.jsonl", "w") as out:
        for r in records:
            out.write(json.dumps(r) + "\n")
    print(json.dumps(summary, indent=2))
    if failures:
        print("DECISION: block", *failures, sep="\n  ")
        sys.exit(1)
    print("DECISION: pass")


RULE = {"max_drop": 0.01, "must_be_perfect": ["adversarial"]}

import os
import sys
from pathlib import Path

import anthropic
from pydantic import BaseModel, ConfigDict

from app.claims import triage  # the system under test: the production entry point
from evals.harness import RULE, main  # the sketch above

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY (a CI secret)
JUDGE_MODEL = os.environ.get("CLAUDE_JUDGE_MODEL", "claude-sonnet-5")
RUBRIC = Path("evals/judge-rubric.txt").read_text()


class JudgeVerdict(BaseModel):
    model_config = ConfigDict(extra="forbid")
    faithfulness: float
    completeness: float
    tone: float
    evidence: str


def call_judge(case: dict, reply_text: str) -> dict:
    response = client.messages.parse(
        model=JUDGE_MODEL, max_tokens=400, system=RUBRIC,
        messages=[{"role": "user", "content": (
            f"POLICY EXCERPT:\n{case['policy_excerpt']}\n\n"
            f"CANDIDATE REPLY:\n{reply_text}\n\nReturn the verdict JSON.")}],
        output_format=JudgeVerdict,
    )
    if response.stop_reason != "end_turn":  # never score a cut-off or refused verdict
        raise RuntimeError(f"judge stopped: {response.stop_reason}")
    verdict = response.parsed_output
    for name in ("faithfulness", "completeness", "tone"):
        if not 0.0 <= getattr(verdict, name) <= 1.0:
            raise ValueError(f"judge score out of range: {name}")
    return verdict.model_dump()


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2], run_system=triage.run, call_judge=call_judge, rule=RULE)

Match the code to the objectives. score_case is the mixed methodology. It does a deterministic label check and a retrieval hit check. It calls the judge only when the case asks for it. It checks the gate on adversarial cases. aggregate is the per-category reporting. decide is the rule written in advance. main writes the same record shape that the observability pipeline uses, so you can compare offline and online numbers.

The wrong choice and its failure. A harness that only prints a score is just a dashboard. A harness that grades with one method cannot see what the other three graders would catch. A harness with no baseline to compare against cannot detect a regression. Cards: mixed-method-evaluation · evaluation-dataset.

Exam lens — how CCAR-P asks about this domain

Reading the stem: the phrases that show which objective is being tested, and the words the explanation uses.

Clues in the stem → objective being tested:

• “reports 94% accurate” / “thumbs-up rate” / “which metric best…” → Define evaluation metrics. The correct option names a dataset, a scoring rule, and a per-category or p95 breakdown. Wrong options offer more of the same weak measurement, or a metric from the wrong family.
• “the eval keeps passing while production regresses” / “which TWO changes to the dataset…” → Design evaluation datasets. Correct options add cases taken from failures and adversarial cases, and they lock a held-out slice. Wrong options make the same easy source bigger, or replace humans with a judge to save money.
• “version B scores higher overall … should they ship?” → A/B testing and iteration. The correct option has a threshold written in advance, guard metrics for every slice, safety at 100%, and a canary. Wrong options ship based on “overall”, on someone’s approval, or after re-grading until the grader agrees.
• “after a document refresh … model and latency unchanged … confident but wrong” → Diagnose, retrieval first. Wrong options name components that a refresh cannot affect: weights, temperature, context window.
• “after moving to the faster tier … only the complex cases” → Diagnose, model mismatch. The fix is to route the hard slice back to the larger tier, not to rewrite the prompt for everyone.
• “format drifts after the prompt edit, on all cases” → Diagnose, prompt failure. Compare the prompt versions; look for instructions that conflict.
• “cost is too high … which change without losing quality” → Optimize. The correct option pairs a lever with its guard metric (caching + hit ratio, tiering + per-category eval, batch + SLA window). Wrong options pull a lever without checking anything: cut the policy text short, move everything to a smaller tier, lower max_tokens without watching stop_reason.
• “which signal would have caught it earliest” / “what must every call log” → Monitor. The correct option is a structured event for each call. It holds versions, retrieval ids, usage, stop_reason, gate decisions, and an alert on a computed ratio. Wrong options are monthly spend, average latency, or logging raw messages.

Trade-off words the explanation uses. Learn to say them the way the answer key does:

Multiple-response items in this domain (questions with two correct answers) usually pair two things. Either a dataset fix with a process fix (add adversarial cases + lock a held-out slice), or a lever with its guard (cache the prefix + alert on the hit ratio). If one of the two answers you choose is a guard for the other, you are probably right.

Related

• Foundations: Domain 5 · Context management & reliability · Observability — explainable, auditable, replayable · Model version pinning & change management · Confidence calibration & stratified sampling
• Foundations: Domain 3 · Prompt iteration as engineering · Message Batches — the cost/latency lever · Millions of PDFs into RAG
• Foundations: Domain 1 · Evaluation of agents · Model portability · Domain 4 · Tokens versus money
• Scenario: S1 · Customer support agent · Deployment: the support agent on GCP
• Professional siblings: Pro 1 · Solution design · Pro 2 · Models, prompting & context · Pro 3 · Integration · Pro 5 · Governance, safety & risk · Pro 6 · Stakeholder & lifecycle · Pro 7 · Enablement
• Hub: Professional track · Glossary cards for this page: evaluation-metrics · evaluation-dataset · mixed-method-evaluation · ab-testing · failure-diagnosis · hallucination-vs-retrieval · model-mismatch · cost-performance-tuning · observability-tooling