Evaluation results

Routing results

Each query has eight candidate intents. SmallDecide scores 97.2%, with a 161 ms median and 222 ms p95. Allocating $0.50/hour to a fully utilized M4 Max gives $0.02307 per 1,000 decisions.

Fable 5, GPT-6 Astra, and DeepSeek V4 Pro score higher, at 99.4–99.8%, with longer API response times. Across the full panel, Llama 3.1 8B is fastest at about 100 ms (96.0% accuracy); Command R7B is cheapest at $0.00482 per 1,000 (97.0%).

Downloads: 30-model JSON, CSV; three-model extension JSON, CSV. Comparison charts.

Confidence gate

The gate accepts SmallDecide’s answer when its maximum candidate probability is at least the fixed threshold. Otherwise it uses the hosted model. On this test, 490 of 500 queries pass locally; ten need a fallback.

These are replays of recorded requests. All 500 queries were sent to every API during measurement. Accepted queries use their local duration; escalated queries add local and hosted durations. The table estimates a serial gate’s cost and latency, without measuring live queueing or deployed savings.

Threshold and errors

The threshold, 0.8502035140991211, was selected on 200 validation cases before the original test. It accepted 193 validation cases with one error. The test accepted 490 cases with 8 errors: 1.63% accepted-case error, above the 1% validation target. The threshold was not retuned.

The eight accepted errors never reach the fallback. Even a perfect fallback would cap this replay at 98.4% accuracy. Improving that ceiling requires changing the local model or acceptance policy, then testing again on new data.

“Gained” and “lost” count individual cases the gate corrects or breaks relative to the direct API. The results do not establish equivalent accuracy. That claim would require a predefined error margin and a paired analysis with enough independent cases.

Gate method · 30-model gate results · Extension records and replay code

Cost & latency

The local run used 166.105 warm compute seconds per 1,000 decisions. For an hourly allocation h and inference utilization u:

Fallback cost uses the actual charges for the ten escalated queries, scaled to 1,000 decisions. Multiplying average API cost by 2% would miss differences in their token bills. At 10% utilization, the allocated local cost is ten times the fully utilized estimate.

Excludes setup, operations, training, and error remediation on both paths; also excludes local model loading and API purchase fees and tax.

Timing conditions

Local timings include tokenization and synchronized warm inference. API timings include network travel, provider scheduling, and generation. Provider GPU types are undisclosed, so this is not an M4 Max versus B200 hardware comparison. Brief ERP probes also ran on the workstation during part of the three-model extension.

With 98% of queries accepted locally, the gate’s p95 can fall entirely within the local path and hide fallback delays. A production test should measure fallback latency, p99, queueing, and cold starts under a stated arrival schedule. Request latency alone does not establish throughput.

Candidate count

Each candidate repeats the state in its prompt. Compute therefore grows with the candidate count even though no output tokens are generated. These H100 timings use different prompts from the M4 Max test and cannot give a hardware speedup ratio.

Latency and token accounting · Costs and client conditions

Model

The backbone is Qwen3-0.6B, with graph modules after layers 9 and 18. Each candidate is encoded with the state and question. Its score is the margin between the Yes and No logits, normalized using a temperature fitted on validation data.

Choice returns the highest-probability candidate. Score returns the expected zero-based rubric index, Σ_i i · p_i. Noul returns the probability of the positive statement.

Confidence

The API’s confidence field measures distribution concentration: 1 − H(p) / log K. It is not a calibrated probability of correctness. The routing gate uses maximum candidate probability, a different quantity.

Graph modules

Each module has 16 learned nodes of width 128, four incoming neighbors per node, and two message-passing rounds. They read prompt representations and add a residual to the final decision token. The nodes are latent slots, not extracted business entities. Graph parameters and rank-16 LoRA adapters were trained together and merged into the release.

Removing graph message passing left greedy success rates unchanged in the reported native-environment tests. That ablation did not demonstrate a benefit from message passing.

Code: candidate scoring, API outputs, graph modules (repository access required). Ablation results.

Test setup

The release report covers 6,217 test and held-out decisions across 23 task families, including identified regression cases reused from earlier evaluations. This is a coverage count, not a pooled accuracy estimate.

Training and splits

Training used 28,234 prepared decisions from public datasets, verifier-labelled tasks, and redacted local-agent execution examples. Validation selected supervised step 1,400 and RL update 10; separate validation groups fitted temperatures. Split sizes and source hashes are in the release report. Private session rows are not redistributed.

The routing study used 200 validation and 500 test queries, excluding source training text, prepared local split groups, and CLINC groups from the release evaluation. Each query contains the correct intent and seven randomized distractors. SmallDecide was trained on CLINC training data; this test measures eight-way in-scope selection, not full 151-class classification or out-of-scope detection. Foundation-model pretraining exposure is unknown.

API conditions

Original panel · 30 models

Reasoning disabled where supported. One attempt per case, four concurrent requests per model and 24 globally. Default provider routing. The protocol records unavailable entries and the preflight amendment.

Extension · three models

Fable 5, GPT-6 Astra, DeepSeek V4 Pro. Low reasoning effort, 4,096-token reasoning/output cap, four concurrent requests globally. Reuses the already-inspected test set; exploratory results.

All APIs were asked for one compact label. Exact intent-name normalization was accepted across models; numeric-key compliance was recorded separately. Failed requests count as incorrect.

Original protocol · Splits and records · Extension protocol · Release report data. CLINC: Larson et al. (2019), CC-BY-3.0.

Factorio

Two identical circuit factories receive twelve scripted proposals at the same ticks. One executes all of them; the other executes only SmallDecide’s allow decisions. The game’s furnaces, assemblers, belts, and inserters produce the circuits. The runner inserts no finished circuits and does not write production values.

Watch the production shift 1 min 41 sec

SmallDecide blocks pole removal before a backup connection exists and allows it after installation. Final dispatch: 583 circuits with review, 482 without. The reviewed factory keeps its 200-plate maintenance reserve and first marks the shipment complete with 558 circuits loaded.

What the model receives

The adapter traverses power and belt connections and supplies textual preconditions. SmallDecide answers action-specific questions over those facts. It does not discover connectivity, plan the factory, or read video frames. The baseline is an execute-all script, not another model.

The film replays recorded decisions at 4× game time. Every intervention’s before/after state and tick matches the original run. The scenario was developed with the model; it is not a held-out reliability test. The first run’s mistakes and subsequent input probes are included in the downloads.

Original trace · Replay states · First run · Source · Reproduction

Audit tasks

In the release’s session execution forecast task, the default threshold detects 6 of 142 failures (4.2% recall). Always predicting success would score 89.0% accuracy. For this task, fault recall and false alerts are more useful than aggregate accuracy.

Each task’s 48 cases are twelve wordings with four identifier variants, from six paired scenario families. A four-prompt search produced no task meeting the frozen 95% selection target. The separate holdout was left unevaluated.

Proposed follow-up

The next protocol would hold out scenario and template families, include deterministic rules and cheap-model gates, and freeze thresholds before test. It would measure fault recall, false alerts, unresolved cases, paired errors, total cost, and p95 workflow delay. Its targets of 99% fault recall and ≤5% false alerts have not been achieved.

Even those targets can create a large review queue. At an assumed 1% fault rate, 10,000 events would yield 99 caught faults and 495 false alerts: 16.7% alert precision. This is a hypothetical calculation, not a measured SmallDecide result.

Exact thresholds, booleans, and inventory rules also need a deterministic-code baseline. The Factorio adapter already computes connectivity; a useful follow-up must isolate what the learned reviewer adds.

Failure confusion matrix · ERP inputs and results · Proposed protocol

Reproduce

The extension bundle includes inputs, local and hosted records, manifests, the parser, and replay code. The command below verifies hashes, recomputes the validation threshold, and checks direct and gated metrics. It needs Python 3.10+ and httpx; no weights, credentials, or paid API calls.

curl -fLo routing-evidence.zip \
  https://smalldecide.com/evidence/routing-frontier-20260917/evidence.zip
unzip -q routing-evidence.zip -d routing-evidence
cd routing-evidence
python3 -m venv .venv
.venv/bin/python -m pip install httpx
.venv/bin/python tools/reproduce_routing_frontier.py --source .

Rerunning inference requires the checkpoint and a compatible runtime; hosted calls also need credentials and a spending limit. Factorio requires a licensed 2.0.73 installation. Its archive contains no game assets or weights. Shorter examples are also available.

Checkpoint & sources

Routing and Factorio results use the v0.3.0 checkpoint with SHA-256:

The public-data checkpoint (f64a7763…) has a separate report. Its results are not included here. Tables are generated from source JSON; the build verifies source and media hashes.

Source hashesRelease dataRelease PDFPublic-run PDF

Evidence downloads are public. Repository and checkpoint downloads require GitHub access. Data attribution and code licensing are recorded in the bundles. Factorio is made by Wube Software; this is an independent demonstration.

Related: training and evaluation guide.