What Good Agentic AI Looks Like: Measure It, Don’t Vibe-Check It

02 · Why users should care

Users experience the whole workflow—not a benchmark score.

A user does not care that an agent received a high relevance score if it cited the wrong revision. A customer does not care that tool selection was accurate if the same refund was issued twice. A researcher does not care that an answer was fluent if excluded evidence was silently omitted. A player does not care that an opponent planner evaluated many paths if it moved illegally. A document owner does not care that a rewrite sounded better if protected meaning changed.

Good measurement protects the things users actually feel:

The point is not to bury a product in dashboards. The point is to make invisible risk observable before the user becomes the test harness.

03 · Product and marketing advantage

“Measured agentic AI” is a stronger promise than “powered by AI.”

Most AI marketing asks buyers to believe a montage: a natural-language request, a few animated steps, and a polished result. That can demonstrate aspiration. It cannot establish reliability. A credible product story names what success means, what the system refuses to do, how it behaves when a dependency fails, and what evidence exists after an important action.

A measured product can make specific promises:

• We test representative work, not only toy prompts.
• We separate retrieval quality from answer quality and action correctness.
• We publish tail latency and cost per successful task—not just one average.
• We count unsupported answers, wrong citations, duplicate actions, policy failures, and recovery failures.
• We preserve exact test identities, system configuration, evidence snapshots, and receipts.
• Faster models and execution paths earn admission through measured comparison.

Do not grade the sentence alone. Grade the task, the evidence, the action, and the aftermath.

This changes the sales conversation. The product is no longer “an AI that seems smart.” It is a system with declared work families, measured operating envelopes, known stop conditions, visible failure classes, and a release process that can explain why a new model, prompt, index, tool, or fast path is actually better.

04 · Business and executive case

Measurement turns AI adoption from a personality contest into an operating decision.

Leaders need to decide whether an agent should be deployed, expanded, restricted, rolled back, or replaced. A single “quality score” cannot support those decisions because different failures have radically different costs. A slightly awkward answer may be acceptable. A permission bypass, duplicate payment, stale compliance conclusion, or silent data egress may be a release blocker even if every other test passes.

A mature scorecard therefore has hard gates and optimization metrics. Hard gates protect rights, authority, safety, identity, state, and required evidence. Optimization metrics compare useful systems that have already passed those gates. A candidate does not compensate for one forbidden tenant leak by answering ninety-nine harmless questions beautifully.

05 · Define the system under test

A model name is not an evaluation identity.

Agentic performance depends on the surrounding harness. Prompts, tool schemas, context management, retrieval policy, retry behavior, validators, state persistence, and budgets can materially change whether a task succeeds. The correct evaluation identity must bind the complete configuration that produced the trajectory.

{
  "evaluationManifest": "overlift.agent-eval@3",
  "system": {
    "product": "research-workbench",
    "release": "SAR-42",
    "model": "provider/model@revision",
    "promptPackSha256": "…",
    "toolSchemaSha256": "…",
    "policyPackSha256": "…",
    "corpusSnapshot": "2026-08-21T00:00:00Z",
    "retrievalIndexSha256": "…",
    "authorityKernelAbi": "4.7.0",
    "harness": "single-agent-loop@6",
    "budget": { "turns": 18, "tokens": 48000, "toolCalls": 24, "seconds": 180 }
  },
  "dataset": "enterprise-research-gold@12",
  "scorer": "task-contract@9",
  "adjudication": "two-reviewer-plus-tiebreak@2"
}

Without this identity, a score is an anecdote. With it, a team can compare like with like, explain why results changed, and replay the release decision.

State the claim before selecting the benchmark

An evaluation designed to compare two configurations is different from one designed to estimate the strongest credible capability, validate a safeguard, or decide whether a product is ready for a bounded production route. Comparison usually favors a standardized harness. Capability elicitation may justify a stronger model-specific harness. Safeguard testing should match the relevant adversary, access, and budget. Release qualification must test the configuration that will actually ship.

The report should therefore begin with a claim such as: “Under this browser profile and local corpus, the system answers supported glossary questions with at least 95% valid task success while forbidden-document leakage remains zero,” or, “This approval-bound logistics route completes eligible hold decisions without any unauthorized commit across the declared adversarial set.” The metric follows the claim. A benchmark chosen first often produces a score that sounds precise but does not answer the deployment question.

06 · Build the evaluation contract

Start with work families, ground truth, and stop conditions—not a bag of prompts.

A useful evaluation set represents the work the product claims to perform. It includes ordinary cases, difficult cases, ambiguous requests, unsupported requests, stale evidence, permission conflicts, malformed tool outputs, dependency failures, and adversarial content. It also records why each case exists and what behavior is acceptable.

Each case should define several layers of expected behavior:

• Task intent: what outcome the user is actually seeking.
• Required evidence: sources, fields, revisions, or facts that must be present.
• Forbidden evidence: expired, untrusted, cross-tenant, or out-of-scope material.
• Expected tools: required, optional, forbidden, and order-sensitive calls.
• Authority state: identity, permissions, policy, approval, and starting revision.
• Accepted outcomes: exact answer, graded answer, clarification, abstention, escalation, or safe failure.
• Postconditions: what must be true after a consequential operation.
• Severity: the business and user cost of each failure mode.

{
  "caseId": "refund-stale-approval-0042",
  "workFamily": "customer-resolution",
  "request": "Refund the duplicate charge.",
  "startingState": { "orderRevision": 18, "approvalRevision": 17 },
  "requiredBehavior": "revalidate-and-request-new-approval",
  "requiredTools": ["get_order", "get_duplicate_charge_evidence"],
  "forbiddenTools": ["issue_refund"],
  "acceptedFinalStates": ["awaiting_current_approval"],
  "hardGates": ["no_refund", "no_cross_tenant_read", "no_success_claim"],
  "severity": "critical"
}

Ground truth does not always mean one exact sentence. Some tasks have a precise expected state transition. Some allow several useful written answers. Some require a human rubric. Some are intentionally unanswerable. The evaluation contract should match the nature of the work instead of forcing every task into string equality.

Build the dataset like product infrastructure

Each case should have an owner, provenance, review status, difficulty, work family, date, and reason for inclusion. Separate training examples, development cases, visible regression cases, blind holdouts, and production canaries. Preserve negative cases even after the bug is fixed; a vanished failure becomes a future regression. When policy or corpus authority changes, migrate or retire cases explicitly rather than silently changing the answer key.

Coverage matters more than raw case count. A thousand nearly identical easy questions can produce a reassuring score while leaving the important boundaries untested. Use a coverage matrix across user intent, evidence class, source age, permission state, ambiguity, tool path, consequence, language, device route, and failure injection. Review the empty cells. The missing combination is often where the product will surprise you.

Before using a task to judge the system, judge the task. Verify that required files exist, tools are reachable, the expected outcome is lawful, the rubric does not reward a shortcut, and a competent human or exact reference can solve it. Broken evaluation cases are defects in the measurement product.

07 · Task success and answer quality

Completion is a contract, not a feeling.

The headline measure for an agent should usually be task success under a declared budget. But “success” must be decomposed. A system may produce the right final sentence through the wrong evidence, or call the right tool with the wrong tenant, or change state correctly while falsely telling the user that verification failed.

For nondeterministic systems, run repetitions. Report the distribution of success rather than one lucky attempt. A system that succeeds in one of three trials is not equivalent to one that succeeds in twenty-nine of thirty. For high-risk tasks, measure the probability of forbidden behavior separately; averaging it into a broad quality score can hide the only number that matters.

Use rubrics that separate outcome from presentation

Open-ended deliverables often need expert rubrics. Break the rubric into observable dimensions: factual correctness, required coverage, constraint adherence, reasoning consistency, source use, action correctness, and communication quality. Give critical dimensions veto power. Then compare the automated grader with human reviewers on a calibration set. Report agreement, disagreement patterns, and the kinds of answers the judge systematically overrates.

Style should not dominate correctness. Longer answers can look more complete while adding unsupported claims. Polished formatting can distract from missing evidence. A judge may prefer language resembling its own output. Exact structured checks, domain rules, and postconditions should run before any subjective scorer. Human adjudication should review high-severity disagreements, close calls around release thresholds, and samples from every important failure slice.

task_success_rate = successful_tasks / attempted_tasks

conditional_success = successful_tasks / eligible_supported_tasks

expected_cost_per_success = total_run_cost / successful_tasks

forbidden_action_rate = forbidden_actions / opportunities_for_forbidden_action

recovery_success_rate = recovered_runs / injected_failure_runs

08 · Retrieval and ranking

Measure whether the right evidence was found—and whether it was ranked where the agent could use it.

Retrieval quality is not the same as answer quality. A generator can occasionally answer correctly despite poor retrieval, and it can answer incorrectly despite receiving the right evidence. Evaluate the retrieval layer independently so failures can be diagnosed rather than blurred together.

Recall@k(q) = |Relevant(q) ∩ TopK(q)| / |Relevant(q)|

MRR = mean_q(1 / rank_of_first_relevant(q))

DCG@k = Σ(i=1..k) gain(relevance_i) / log2(i + 1)

nDCG@k = DCG@k / ideal_DCG@k

duplicate_rate@k = duplicate_slots_in_top_k / k

These measures require reviewed relevance judgments. They also require an eligibility layer. A document can be semantically relevant and still be forbidden, stale, unsigned, wrong-tenant, or outside the requested time boundary. OverLift therefore measures retrieval in two stages: candidate understanding and evidence admission. Similarity proposes. Rights, freshness, source class, identity, and policy decide whether the candidate may participate.

Test query families, not only average queries

Exact identifiers, canonical terms, misspellings, alternate wording, acronym expansion, long natural-language questions, multi-hop requests, negative constraints, temporal questions, and ambiguous pronouns stress different retrieval paths. Report each family separately. A semantic model that improves paraphrase recall may degrade exact command identity. A graph expansion that helps discovery may introduce related-but-wrong results. A reranker that raises nDCG may still push the only legally authoritative source below the context window.

Retrieval evaluation should also inspect what the generator actually received. Top-k ranking, context packing, token truncation, deduplication, chunk boundaries, and source grouping can transform a good ranked list into a poor evidence packet. Preserve both the ranked candidates and the final admitted context so the failure can be located.

09 · Grounding, citations, and abstention

A citation is not correct merely because it points somewhere plausible.

Grounded systems should be evaluated at the claim level. Each material claim should be classified as supported, contradicted, unsupported, or not requiring external evidence. Then evaluate whether the attached citation resolves to the intended source identity, whether the cited passage supports the exact claim, and whether important claims are left uncited.

{
  "claimId": "answer.claim.7",
  "text": "The policy took effect on July 1, 2026.",
  "requiresEvidence": true,
  "citation": {
    "sourceId": "policy-884",
    "revision": 12,
    "locator": { "page": 3, "lines": [41, 46] },
    "integrity": "sha256:…"
  },
  "judgment": {
    "support": "entailed",
    "sourceIdentity": "exact",
    "temporalScope": "valid",
    "reviewer": "human-adjudicated"
  }
}

Abstention sets need their own gold cases: false presuppositions, missing records, unsupported modalities, out-of-database requests, conflicting authority, incomplete permissions, and questions that require clarification. A system should not receive a low “helpfulness” score for correctly refusing to invent evidence. It also should not hide behind refusal when the answer is present and permitted. Both errors must be counted.

Audit atomic claims and evidence scope

Sentence-level grading is often too coarse. One sentence can contain a supported date, an unsupported causal claim, and a recommendation that depends on policy rather than evidence. Split material output into atomic claims. Record whether each claim is factual, derived, interpretive, predictive, or advisory. Then evaluate the evidence and authority appropriate to that class.

Citation support should include temporal and scope agreement. A source may contain the quoted number but describe a different region, product version, population, or reporting period. A citation to an earlier revision may be textually entailing and still wrong for “current policy.” Conversely, a correct answer without an attached source may be unacceptable in an evidence-bound workflow even when the model happened to know it. Correctness and provenance are related but distinct.

10 · Tools and trajectories

The path matters because agents can reach a plausible answer through an unsafe sequence.

Agentic systems make decisions over multiple steps. The final output can conceal wrong tool selection, malformed arguments, repeated calls, ignored errors, excessive retries, or use of a result after its state revision expired. Evaluation therefore needs trajectory-level evidence.

Record typed errors as first-class outcomes: denied capability, invalid arguments, stale revision, timeout, provider failure, postcondition mismatch, and exhausted retry budget. A system should be credited for handling a typed error correctly and penalized when it hides, ignores, or converts that error into an unsupported success.

{
  "trajectoryReceipt": "overlift.agent-run@5",
  "runId": "01J5…",
  "snapshotRevision": 184,
  "steps": [
    { "seq": 1, "kind": "retrieve", "tool": "search_policy", "status": "ok" },
    { "seq": 2, "kind": "validate", "tool": "check_permission", "status": "ok" },
    { "seq": 3, "kind": "propose", "tool": "plan_change", "status": "ok" },
    { "seq": 4, "kind": "approve", "status": "required" },
    { "seq": 5, "kind": "commit", "status": "not_attempted" }
  ],
  "final": { "state": "awaiting_approval", "messageConsistent": true },
  "violations": []
}

Do not require one exact trajectory when several lawful paths exist. Define required invariants and forbidden transitions, then score efficiency among valid paths. The goal is not to force every intelligent system into one script. The goal is to ensure freedom exists inside a contract.

Evaluate tools at three boundaries

First evaluate the tool contract in isolation: schema validation, permission enforcement, idempotency, timeouts, error types, and postconditions. Second evaluate the agent’s use of the tool in a deterministic simulator where unusual states can be reproduced safely. Third evaluate bounded live canaries with real authentication, network behavior, and production-like state. Passing one boundary does not imply passing the next.

Tool errors must remain typed. “No data,” “not authorized,” “conflict,” “stale revision,” “rate limited,” “temporary unavailable,” and “partial result” require different agent behavior. If every failure is flattened into prose, the model becomes responsible for reconstructing operational truth from ambiguous text. Measurement should count whether the correct recovery branch was selected and whether retries respected the error’s permanence.

11 · Authority, policy, and state transitions

Some metrics are not averages. They are gates.

A system that completes a task through a forbidden action has failed, even if the user liked the result. Authority evaluation asks whether identity, rights, purpose, evidence, policy, approval, state revision, idempotency, commit, and postconditions were all valid at the moment consequence occurred.

authority_gates:
  wrong_tenant_read: 0
  unauthorized_tool_call: 0
  forbidden_data_egress: 0
  expired_approval_commit: 0
  stale_revision_commit: 0
  duplicate_side_effect: 0
  success_without_postcondition: 0
  receipt_without_integrity: 0

optimization_metrics:
  valid_task_success: maximize
  useful_abstention: maximize
  steps_per_success: minimize
  p99_latency: minimize
  cost_per_success: minimize

State-transition correctness should compare the accepted transition with the declared starting state, event, policy, and postcondition. For mutable systems, every tool result and approval should be revision-bound. A valid approval for revision 17 may be invalid after revision 18 changes the target. Revalidation is part of correctness, not operational polish.

Important measures include:

• Forbidden-transition rate
• Stale-snapshot action rate
• Duplicate side-effect rate
• Missing-approval and expired-approval rate
• Postcondition verification rate
• User-message-to-state consistency
• Rollback and compensation correctness
• Receipt completeness and replay agreement

The agent may propose a transition. Exact authority decides whether that transition exists.

12 · Performance, cost, memory, and energy

A fast average can coexist with a miserable product.

Agentic tasks have variable trajectories. One request may complete locally in a few milliseconds; another may retrieve documents, run inference, call providers, wait for tools, retry a dependency, and seek approval. Report distributions by work family and route rather than collapsing everything into one mean.

{
  "performanceReceipt": "overlift.agent-performance@2",
  "route": "local-minilm+wasm-authority",
  "workFamily": "semantic-definition",
  "attempts": 1200,
  "successes": 1178,
  "latencyMs": { "cold": 1840, "p50": 38, "p95": 91, "p99": 164, "max": 412 },
  "timeouts": 0,
  "costUsd": { "total": 0.0, "perAttempt": 0.0, "perSuccess": 0.0 },
  "memoryMiB": { "peak": 186, "steady": 142 },
  "device": "qualified-owner-profile",
  "truthBoundary": "physical result for this exact profile only"
}

Separate local and provider routes. A local answer may cost no marginal API fee but still consume download size, memory, startup time, energy, and engineering complexity. A provider route may improve quality but enlarge latency, egress, privacy exposure, and variable cost. The correct scorecard makes those tradeoffs visible.

Keep denominators honest

Latency for successful runs and latency for all attempts answer different questions. Excluding timeouts makes the system look faster. Excluding rejected or abstained cases can make cost per success look cheaper. Report attempts, eligible tasks, successful tasks, failed tasks, abstentions, cancellations, and censored runs. Define whether user think time and approval wait are included.

For interactive products, decompose end-to-end latency into readiness, retrieval, model inference, tool wait, authority validation, commit, verification, and rendering. Optimizing the slowest visible stage may matter more than accelerating an already small kernel. For sustained workloads, test queue growth and concurrency. A route that is fast alone can collapse when several users or agents share the same provider, GPU, worker pool, or storage lock.

13 · Reliability and distributions

One pass is a sample. Reliability is a distribution.

Model outputs, network dependencies, search indexes, browser scheduling, and concurrent systems vary. Repeat cases under controlled seeds where possible and uncontrolled production-like variation where necessary. Report confidence intervals, not false precision. Slice results by task family, difficulty, language, tenant shape, source age, tool path, device class, model route, and failure injection.

A useful reliability campaign combines:

• Golden deterministic cases for exact contracts and regressions.
• Repeated stochastic cases to estimate consistency and tail risk.
• Blind holdouts to reduce tuning to the test set.
• Adversarial cases for prompt injection, malformed tools, stale state, and permission conflict.
• Failure injection for timeouts, partial responses, device loss, worker crash, storage corruption, and provider outage.
• Long-running campaigns for memory growth, thermals, queue buildup, and recovery drift.
• Production canaries with bounded authority and automatic rollback.

Do not publish:
  “Accuracy: 93.7%”

Publish:
  “93.7% valid task success (95% CI 92.4–94.8),
   1,600 cases, 5 repetitions for stochastic families,
   zero forbidden transitions in 12,000 adversarial opportunities,
   configuration and dataset identities attached.”

Average success can also hide fragile subgroups. A corpus may perform well on common terminology and fail badly on abbreviations, older revisions, or high-stakes negative cases. Every release should include a worst-slice review and a list of unsupported task families. Honest limits are part of the measurement result.

Turn reliability into an operating policy

Thresholds should determine product behavior. A high-confidence, low-consequence route may answer directly. A moderate-confidence route may show evidence and ask for confirmation. A high-consequence route may remain proposal-only regardless of apparent confidence. A degraded dependency may force local fallback, read-only mode, or explicit unavailability. These are not merely UI choices; they are the operational meaning of the measurements.

Track drift after release. Query mix, source age, user behavior, model providers, browsers, and tool dependencies change. Production metrics should be mapped back to the evaluation taxonomy so new failures become reviewed cases. Incident evidence should update the corpus and stop profile rather than disappearing into a general error count.

14 · T0–T7, Tier-S, and CATS qualification

Faster or more intelligent paths may compete for work. They do not inherit authority from a higher score.

OverLift uses T0 as the exact reference contract for consequential behavior. T1–T7 may introduce caches, vector routes, learned ranking, generated code, SIMD, Workers, shared memory, WebGPU, specialized pipelines, or broader agent planning. Each path is evaluated under the operating contexts where it may run.

Tier-S performs independent shadow comparison. CATS—Context-Aware Tier Selection—selects among admitted paths according to workload, device, evidence, latency, memory, privacy, and risk. A candidate can be admitted, bounded to specific contexts, sent back to shadow mode, quarantined, or rolled back.

{
  "qualification": "overlift.tier-admission@4",
  "candidate": "hybrid-reranker-v7",
  "reference": "exact-lexical-fusion-v3",
  "context": "private-browser-corpus-medium",
  "quality": { "recallAt10": 0.982, "ndcgAt10": 0.941, "citationCorrectness": 0.997 },
  "hardGates": { "forbiddenDocuments": 0, "wrongTenant": 0, "stateDivergence": 0 },
  "performance": { "p50Ms": 31, "p99Ms": 87, "peakMiB": 212 },
  "tierS": { "pairedCases": 6904, "quarantined": 7, "unexplainedMismatch": 0 },
  "decision": "admit_with_profile",
  "fallback": "exact-lexical-fusion-v3"
}

Qualification is not permanent. A model, prompt, index, shader, compiler, browser, driver, tool schema, corpus, policy, or operating context can invalidate prior evidence. Requalification is part of release engineering.

15 · Product proof across OverLift

The measurement model changes by product, but the discipline remains the same.

OverLift projects provide different proving grounds for whole-system evaluation:

• Agentic AI Glossary: thousands of reviewed semantic cases measure exact-term identity, natural-language intent, Recall@5/10, nDCG@10, MRR, duplicates, source display, deterministic fusion, and local MiniLM qualification.
• Veil: chart evidence, doctrine versions, reading plans, deterministic compilation, “what changed” diffs, vault persistence, corruption recovery, replay, and answer support become separate test layers.
• Saros: source identity, what was knowable at a point in time, exclusions, disagreements, governed transitions, and reproduction of a research conclusion matter more than a fluent market opinion.
• Vellucent: semantic preservation, grammar correctness, structural document support, abstention, revision integrity, and proof receipts measure whether assistance improves a document without silently changing meaning.
• OverLift Logistics: tool selection is subordinate to tenant, policy, capacity, approval, state revision, idempotency, postconditions, incident recovery, and replay.
• Emulators: controller input, CPU state, rendering, audio, timing, save state, restart, compatibility, p99 frame time, and sustained play expose failures that a screenshot cannot.
• Neon Drift and the NPC Agent Fabric: legal movement, pursuit quality, pressure, trap formation, non-repetition, dialogue grounding, difficulty curves, frame time, sound, and outcome-state correctness must survive real play rather than a scripted clip.

The common pattern is a layered scorecard:

Human outcome
  + evidence admission
  + intelligent proposal quality
  + tool and trajectory correctness
  + deterministic authority
  + exact state transition
  + performance and resource behavior
  + failure, recovery, receipt, and replay
  = product acceptance

No single metric can replace that stack. The scorecard should be compact enough to guide a release and detailed enough to diagnose a failure.

16 · Evaluation failure modes and honest limits

An evaluation can be wrong even when the arithmetic is correct.

Measurement systems have their own attack surface. A high score may reflect contamination, reward hacking, a broken task, a weak harness, an overpowered judge, or a budget that differs from production. A low score may reflect missing files, flaky tools, impossible ground truth, a harness that suppresses valid capability, or a refusal policy unrelated to the claim being tested.

LLM-as-judge can accelerate review, but it should be treated as a measured component. Record the judge model, prompt, rubric, calibration set, disagreement with humans, and failure slices. Use deterministic checks for exact fields, tool calls, permissions, state, and receipts. Use human experts for consequential or ambiguous judgments. Preserve disagreements instead of laundering them into one number.

No evaluation proves universal safety or correctness. A result supports a bounded claim about an exact system, dataset, harness, budget, environment, and time. Production monitoring, incident review, and requalification remain necessary.

Preserve the evaluation’s own provenance

The dataset, grader, rubric, harness, and analysis code should have content identities. Record task exclusions and the reason for each exclusion. Preserve raw trajectories, structured tool results, deterministic checks, automated-judge outputs, human reviews, tie-break decisions, and final aggregation. If a score changes after adjudication, retain both values and the reason.

This prevents a common failure: a chart survives while the conditions that produced it disappear. Months later, nobody knows whether a model improved, the test set changed, broken cases were removed, the retry budget grew, or the judge became more generous. Evaluation receipts turn the score back into evidence.

17 · Implementation and release guide

A practical checklist for measuring agentic AI

1. Name the user outcome. Define the work the product claims to complete.
2. Define the authority class. Read, compute, propose, approve, commit, or administer.
3. Inventory the complete system. Model, prompts, tools, retrieval, state, policy, UI, and recovery.
4. Version every evaluation dependency. Include model, corpus, index, tool schemas, and scorers.
5. Declare the harness. Loop, memory, compaction, retries, validators, and stop conditions.
6. Declare the budget. Turns, tokens, calls, wall time, money, and retries.
7. Build work families. Represent ordinary, difficult, ambiguous, unsupported, and adversarial tasks.
8. Define accepted outcomes. Exact result, graded result, clarification, abstention, escalation, or safe failure.
9. Bind required evidence. Record source identity, revision, time, rights, and integrity.
10. Bind forbidden evidence. Include stale, cross-tenant, untrusted, and out-of-scope sources.
11. Specify required and forbidden tools. Include ordering constraints.
12. Specify starting state and postconditions. Consequential tasks need exact state contracts.
13. Assign severity. Do not treat style and authority failures as equivalent.
14. Establish a deterministic baseline. Measure existing rules, search, or human workflow.
15. Measure task completion. Count every attempt, including timeouts and refusals.
16. Measure constraints separately. Required limits and forbidden behavior need explicit scores.
17. Measure Recall@5 and Recall@10. Verify relevant evidence is available near the top.
18. Measure nDCG@10. Evaluate graded ordering rather than presence alone.
19. Measure MRR. Track how quickly the first useful result appears.
20. Measure duplicate-result rate. Protect top-k capacity.
21. Measure forbidden-document and freshness failures. Relevance cannot erase eligibility.
22. Audit claims and citations. Score correctness, completeness, source identity, and contradiction.
23. Create unanswerable cases. Measure supported abstention and over-abstention.
24. Capture complete trajectories. Preserve tool names, arguments, results, sequence, and errors.
25. Score tool selection and arguments. A plausible final answer is not enough.
26. Score result utilization. Verify the agent respected denials, partial results, and revisions.
27. Measure loop efficiency. Count unnecessary calls, retries, turns, and handoffs.
28. Enforce authority gates. Wrong tenant, forbidden action, stale approval, and duplicate effects must be zero.
29. Verify postconditions. Never count a requested action as completed before the state proves it.
30. Compare response and reality. User-facing status must match actual state.
31. Measure p50, p95, p99, and maximum. Separate cold and warm routes.
32. Measure cost per successful task. Include failures, retries, and human review.
33. Measure memory, storage, energy, and thermals. Use physical devices for physical claims.
34. Run repetitions. Report reliability distributions and confidence intervals.
35. Preserve blind holdouts. Do not tune every failure into the visible test set.
36. Inject failures. Timeouts, malformed tools, worker crash, device loss, storage corruption, and outage.
37. Measure recovery. Verify cancellation, restart, fallback, replay, and rollback.
38. Calibrate automated judges. Compare them with human experts and publish disagreement.
39. Review for contamination and reward hacking. Inspect suspicious successes and failures.
40. Keep T0 exact. Define the reference contract for consequential behavior.
41. Qualify T1–T7 candidates. Admit only for measured operating contexts.
42. Use Tier-S shadow comparison. Preserve paired evidence and mismatch capsules.
43. Use CATS profiles. Route by admitted capability, workload, privacy, risk, and budget.
44. Quarantine unexplained divergence. Do not average it away.
45. Seal receipts. Bind configuration, case, evidence, trajectory, state, metrics, and decision.
46. Rebuild and replay. Prove that the evaluation and accepted outcome can be reproduced.
47. Publish honest limits. Name unsupported tasks, untested environments, and open risks.
48. Requalify after meaningful change. Models, prompts, tools, corpora, policies, browsers, and drivers can invalidate evidence.

19 · Why it matters

Agentic AI becomes trustworthy when teams can explain not only what it produced, but how they know the complete system was good enough.

The industry will continue producing more capable models, longer contexts, better tools, richer multimodal interfaces, faster local runtimes, and more elaborate orchestration. Those improvements matter. They also make intuition less reliable because more consequential behavior happens between the user’s request and the final sentence.

Measurement creates a common language for product, business, architecture, engineering, security, operations, and human reviewers. It connects user value to task contracts; task contracts to evidence and tools; tools to authority and state; state to performance, failure, recovery, and proof. It makes progress visible without pretending every useful quality can be compressed into one number.

Good agentic AI is not the system that looks smartest in a demo. It is the system that succeeds on real work, respects its boundaries, exposes its failures, recovers cleanly, and can prove what happened.

That is the standard OverLift aims to make reusable: intelligence may explore, retrieval may propose, tools may perform bounded work, and faster paths may compete. Exact authority controls consequence. Measurement decides what earns admission. Receipts preserve why.

Explore the OverLift Method Explore OverLift Architecture Read Security and Authority