Less than 1/10th the cost.
Over 8× faster decisions.

SmallDecide runs on your own machine. In this 500-case routing replay, its confidence gate handled 490 requests locally and sent ten to an API. Accepted requests need no network call or per-token API charge; local compute is included in the cost estimate.

Putting it ahead of Fable 5, Astra, or DeepSeek Pro brought estimated cost below one tenth of using those APIs alone. Median decisions were over 8 times faster. These savings come from replaying a confidence-based gate over measured local and API responses.

How the gate works

SmallDecide scores the candidate answers. If its maximum candidate probability is at least 0.8502035140991211, the local answer is accepted. This is distinct from the API’s entropy-based confidence field. Otherwise, the request goes to the selected API. We fixed that threshold using 200 separate validation cases before the original test.

The estimate charges for local compute on every request and API calls on escalations. Every test case was sent to every API to establish the baseline; the avoided calls are calculated from those stored responses.

Accuracy trade-off

With Astra, the combined system scores 492/500, versus 498/500 for Astra alone. Estimated cost falls 96.5%; median decision time falls from 1,349 to 162 milliseconds.

The gate accepts some incorrect local answers with high confidence. A production threshold needs validation on the traffic it will handle.

Some small APIs cost less than local inference. Command R7B (12-2024) costs $0.00482 per 1,000 at 97.0% accuracy, compared with SmallDecide’s estimated $0.02307 at 97.2%. Adding the local gate to a cheap API can increase the bill.

The cost estimate depends on how much you use the machine. Adjust its hourly cost and utilization below to compare local inference with an API.

Scope

This test measures request routing. Invoice audits and agent-action checks need separate evaluations. Our ERP development probe did not reach its 95% accuracy target; the proposed audit benchmark has not run.

Methodology

We used 500 frozen, in-scope queries from CLINC by Larson et al. (2019), licensed CC-BY-3.0, at a pinned revision. Each query has its correct intent plus seven randomly selected distractors, shuffled. This is narrower than full 151-label classification and excludes out-of-scope queries.

SmallDecide was trained on CLINC training data. The test excludes upstream training text and examples from the released evaluation archive. Foundation-model training exposure is unknown. The later three-model run reused this test after we had examined the original results, so it is exploratory.

API settings. The original 30-model run used a 128-token output cap, reasoning disabled where supported, and up to 24 requests in flight (four per model). Fable 5, Astra, and DeepSeek V4 Pro used low reasoning effort, a 4,096-token combined reasoning/output cap, and four requests in flight. Each model saw the same prompt and candidate answers. Labels were withheld; failed or unparseable responses counted as incorrect.

Timing. SmallDecide v0.3.0 ran sequentially on an Apple M4 Max using MPS. Timings include tokenization and synchronized inference, but exclude the 8.43-second model load. All other models were tested through hosted APIs; their serving hardware was not reported. API timings include the network and full response. Brief ERP probes also ran on the workstation during part of the API test. The chart reports these observed timings, rather than an isolated hardware comparison.

Cost. API costs are recorded OpenRouter charges, including reasoning tokens and provider discounts. The local estimate uses measured inference time at $0.50 per machine-hour and full utilization. Training, setup, operations, and error remediation are excluded. The displayed comparison uses the 30 APIs with complete billing; all source runs remain available below.

Replay. We reused the maximum-candidate-probability threshold chosen on 200 validation cases. Escalated requests add local time and API time. This estimates the gate’s cost and latency; it does not measure a deployed service or queueing. Accuracy differences and timings are single-run observations, not evidence of equal quality or a universal ranking.

Data and code

Inputs, predictions, charge records, protocols, and reproduction scripts for both API runs.

Fable, Astra, DeepSeek data ↓Original 30-model data ↓

Comparison CSV · Comparison JSON · Gate replay · API protocol · Earlier Haiku experiment

Run SmallDecide locally →