# OpenAuditor versus anthropic/claude-haiku-4.5

500 frozen eight-option, in-scope CLINC intent-routing cases. Separate 200-case validation split chose the escalation threshold. This is a routing benchmark, not proof of general auditing or production support performance.

| Path | Correct | Accuracy (95% Wilson interval) | Median latency | p95 latency |
|---|---:|---:|---:|---:|
| OpenAuditor · Mac M4 Max | 486/500 | 97.2% (95.4%–98.3%) | 161 ms | 222 ms |
| Claude Haiku 4.5 · OpenRouter | 494/500 | 98.8% (97.4%–99.4%) | 789 ms | 1154 ms |
| Hybrid replay | 492/500 | 98.4% (96.9%–99.2%) | 162 ms | 229 ms |

**Actual hosted test charge:** $0.080218, or **$0.1604 per 1,000 cases** at this usage mix.
Recorded API tokens: 67,718 input and 2,500 output. The hosted model returned only a label; explanation tokens were not requested.

The frozen hybrid policy handled **490/500 locally** and escalated **10** in replay. That avoids 98.0% of paid calls in this sample.

## Cost scenarios

Local rates below are explicit assumptions. They include whatever electricity, depreciation or rental costs you assign to an allocated machine-hour. Utilization is the fraction of that allocated time spent doing inference.

| Assumed local $/hour | Utilization | Local $/1,000 | Local savings | Hybrid $/1,000 | Hybrid savings |
|---:|---:|---:|---:|---:|---:|
| $0.10 | 100% | $0.0046 | 97.1% | $0.0079 | 95.1% |
| $0.50 | 100% | $0.0231 | 85.6% | $0.0263 | 83.6% |
| $1.00 | 100% | $0.0461 | 71.2% | $0.0494 | 69.2% |
| $0.50 | 10% | $0.2307 | -43.8% | $0.2340 | -45.8% |

Accepted local decisions contained 8 errors out of 490 (1.63%), above the 1% empirical error target used for validation threshold selection. This run does not establish that production error target.

## Boundaries

Hybrid is a replay of measured predictions and request charges; every test case was actually sent to the hosted model for comparison. Local hardware rates are assumptions, not measured electricity/depreciation or a vendor quote. Measured warm sequential Mac compute is used; idle allocation is represented by utilization. Training, setup, operations, loading, and error-remediation costs are excluded. Lower utilization can eliminate apparent savings.

Mac timings include tokenization, model execution and probability conversion. Hosted timings include network and provider execution, measured with four concurrent requests; these are deployment observations, not identical-hardware speed comparisons. Hybrid latency adds the observed local and hosted durations for escalated cases; it is simulated.

Public benchmark exposure during either model's pretraining is unknown. Source training texts and prior release-evaluation cases were excluded from this sample. The correct intent is always among eight options; this does not measure full 151-class intent detection, out-of-scope detection, or auditing errors.

Validation thresholds target empirical accepted-case error, not guaranteed future accuracy. Read the accuracy results alongside every savings figure.

Checkpoint: `ec83247177a6919c17944093fdd02d3258906dc809c16b326e0dfd879275efeb`. Dataset: `43fc867f7e773012cbf202877a5267d3e496947f96cdee549228b237cb9ff73b`.

Sources: [CLINC dataset](https://huggingface.co/datasets/clinc/clinc_oos), [OpenRouter usage accounting](https://openrouter.ai/docs/api_reference/overview), [Claude Haiku 4.5](https://openrouter.ai/anthropic/claude-haiku-4.5).