# Benchmark design: continuous checks inside an agent workflow Status: designed, not executed. This is the next experiment, separate from the completed CLINC routing comparison and its routing-gate replay. There are no results or claimed winners for this protocol. ## The product question Can a small, local decision model check every agent action and reserve expensive review for exceptions, while meeting a fixed error budget? OpenAuditor's relevant capabilities are short, instruction-conditioned choices and probabilities without autoregressive text output. The released checkpoint's strongest relevant existing evidence is on synthetic completion-evidence checks, permission checks, and constrained workflow decisions. Those results motivate this workload; they do not establish performance on realistic business traces. ## The workload Audit a support agent performing tool-backed operations such as a refund request, a file update, a deployment, or an outgoing notification. Use traces with an explicit request, the proposed operation, a tool result, and the agent's final claim. The auditor checks three things at the points where the evidence exists: 1. **Permission, before execution.** Does this specific action stay within the user's authorization, including destination, amount, and object scope? 2. **Completion, after execution.** Does the tool evidence support the agent's completion claim? Include failed, partial, queued, and successful outcomes. 3. **Next action.** Continue, retry, or request review under the supplied policy. An operation's name alone must not reveal the answer. Include minimal pairs where changing the authorization, tool status, or completion claim changes the correct decision. Include negation, mixed successes, stale results, missing evidence, quoted instructions, and attempts to redirect the auditor in untrusted content. Define missing evidence as requiring review rather than silently assuming success. Use execution-backed traces from a sandbox or independently reviewed traces. Do not describe generated templates as customer traffic. A synthetic pilot can check feasibility but must be labeled synthetic and remain separate from the confirmatory result. Simple arithmetic and structured boolean policies also get a deterministic rules baseline; a model has not earned credit for beating an unnecessarily expensive LLM on work that ordinary code already solves perfectly. ## Systems compared - Each of the same 30 hosted models running the complete checking task. - OpenAuditor alone, with its released weights and typed outputs. - OpenAuditor first, escalating uncertain cases to each hosted model. - A fixed rules baseline, and a cheap hosted model used as the first-stage gate. All systems receive the same evidence available at that point in the workflow. Allow hosted models one compact structured response for independent checks. Require separate calls only when later evidence genuinely depends on an earlier action. Do not request explanations or unnecessary probability arrays just to increase competitors' output-token bills. Where a gate needs confidence, freeze its confidence method and calibration on validation data and give competing gates the same validation budget. ## Split and lock before the comparison Start with a clearly labeled feasibility pilot. It is development data and can inform the final protocol, never the final reported test accuracy. For the confirmatory run, reserve 1,000 validation episodes and 2,000 untouched test episodes, balanced between clean and faulty episodes. Split by underlying scenario, template family, and tool interaction, keeping minimal pairs together. Hold out wording and scenario families, not just random identifiers. Freeze inputs, independently verified labels, model IDs, prompts, thresholds, parsing, retry policy, provider configuration, and cost assumptions by hash before calls. Do not tune the released model on the test or reuse the inspected routing test to claim a fresh result. Freeze the full model panel regardless of pilot ranking. Primary engineering targets: catch at least 99% of faulty episodes and falsely flag no more than 5% of clean episodes. These are proposed product requirements, not a claim that the model meets them or a compliance standard. Select gate thresholds on validation only. Report confidence intervals and failure counts; if the test misses a target, there is no quality-qualified winner. Count an unresolved abstention or failed API call as unresolved, not a correct decision. ## Measure the useful outcome The main plots show **total cost against fault recall**, with false-positive rate and unresolved rate visible, and **p95 workflow delay against fault recall**. Only systems meeting the frozen quality targets can claim the lowest cost or lowest delay at that quality level. Retain all other systems in the results. Also report exact whole-episode correctness, per-check confusion matrices, balanced accuracy, risk versus coverage, calibration of confidence used for escalation, paid input/output tokens, escalation rate, throughput, memory, and paired error changes. Do not let a majority of normal events hide missed faults. Run both an actual online stream and a separate throughput test. Use identical arrival schedules and bounded concurrency, record queueing and cold starts, and vary utilization. Do not infer batch throughput by dividing by median single-request latency. Scaling the number of independent checks is useful only when it represents the workload; multiplying identical requests does not create a new advantage. Include allocated local compute, fallback requests, failed calls, retries, and idle time in the cost. Keep infrastructure estimates and actual API charges visibly distinct. Avoid charging the same idle capacity twice. Include a 5% fault-prevalence projection separately from the balanced test. Call it a scenario rather than a measured production distribution. A real deployment claim requires a representative trace sample and observed traffic volume. ## What a successful result would say “On this held-out agent-checking workload, adding OpenAuditor reduced paid calls, total cost, and workflow delay while meeting the predeclared fault-detection and false-alert targets.” Report the actual quantities, hardware, and comparisons. That would demonstrate the intended advantage: inexpensive checks throughout a workflow, with stronger review reserved for exceptions. It would not establish universal superiority, general financial-audit capability, or savings for every traffic pattern.