# Factorio review tokens, cost and latency

A measured hosted-review comparison using the twelve frozen states, questions
and candidate criteria from the existing Factorio production-shift recording.
Each hosted model receives the same decision information in one JSON prompt and
is asked to return only `allow` or `block`. No explanation is requested. The
scenario policy labels were fixed in the protocol before any calls and are not
included in the requests.

All three selected models and the local reviewer agree with all twelve scenario
policy labels. These are curated examples used to develop the local integration,
not a held-out reliability estimate. Hosted models review saved snapshots; they
do not control the game. This comparison does not measure factory output from
hosted decisions or any planner's token usage.

## Observations across all twelve reviews

| Reviewer | API input tokens | API completion tokens | Median review | API charge / local allocation |
|---|---:|---:|---:|---:|
| OpenAuditor 0.6B | 0 | 0 | 156 ms | $0.000228 estimated |
| Fable 5 | 2,637 | 43 | 3,007 ms | $0.028520 billed |
| GPT-6 Astra | 1,675 | 75 | 1,342 ms | $0.020500 billed |
| DeepSeek V4 Pro | 1,620 | 901 | 1,840 ms | $0.002511 billed |

API completion counts already include reported reasoning tokens: 0 for Fable,
13 for Astra, 877 for DeepSeek. Do not add those again. Paid API tokens avoided
by replacing these twelve calls with the local reviewer are 2,680, 1,750 and
2,521 respectively. This means eliminating the hosted review calls, not
eliminating all computation or reducing a planner's bill by those percentages.

OpenAuditor generates no output tokens. Its recorded input counter is 833 state
and instruction tokens; full prompts repeat inputs for each candidate and add
criteria and formatting. The trace records 3,746 non-padding candidate-token
evaluations across the twelve reviews. Model tokenizers and these accounting
categories differ. Zero hosted tokens does not mean zero local tokens, and
candidate scoring does not guarantee fewer total token evaluations.

## Timing and cost assumptions

API calls were made on 2026-09-17 through OpenRouter, one at a time, interleaving
Fable, Astra and DeepSeek for each case. Low reasoning effort was requested for
all three, with a 4,096-token combined completion cap and default provider
routing. Each case has exactly one attempt; no retries or cases were removed.
`records.jsonl` contains provider identity, label, usage, elapsed duration and
actual billed charge. Reasoning text, credentials and account-linked response
IDs are excluded. The complete experiment cost $0.0515307984.

Hosted durations include the network and complete response. Local durations
reuse synchronized warm M4 Max reviews from the earlier Factorio trace; they
include tokenization, but not model loading. The twelve reviews sum to 1.639
seconds. The film's game clock and 4x playback are not inference timings.
Provider GPU types were not reported. This is not an isolated hardware test.

The local cost estimate allocates $0.50 per machine-hour at 100% utilization:

`cost = recorded review seconds * 0.50 / (3600 * utilization)`

This produces $0.0002276368 for twelve reviews, or $0.0189697 per 1,000 at the
same mix and warm timing. These are compute allocations, not observed hosting
bills or measured throughput. They exclude model loading, training, adapter,
Factorio, operations and error remediation. At 10% utilization the local
allocation is ten times larger. At 1% it is $0.022764 for twelve reviews, above
the recorded Astra and DeepSeek API charges. A lightly used dedicated machine
may cost more than an API; sharing the machine changes that allocation.

## Reproduce

`site/factorio_savings.py` recomputes totals and ratios from the response records
and the original trace during `python site/build.py`. The build verifies all
source hashes and publishes `summary.json`, `comparison.csv` and the raw records.
No API calls are made during a build. The source archive contains these files,
the original local trace, the measurement tool and its helper modules.

To recompute the arithmetic without spending API credits, run from the source
archive root:

```python
import json, sys
from pathlib import Path
sys.path.insert(0, "site")
from factorio_savings import summarize
trace = json.loads(Path("report/factorio-shift-20260917/trace.json").read_text())
protocol = json.loads(Path("protocol.json").read_text())
records = [json.loads(line) for line in Path("records.jsonl").read_text().splitlines()]
print(json.dumps(summarize(trace, protocol, records), indent=2))
```

To repeat the paid calls, install Python 3.10+ and `httpx`, fetch a fresh model
catalog from the official OpenRouter `/api/v1/models` endpoint, and save an
object keyed by the three IDs in `protocol.json`. Use a separate local key file
with `OPENROUTER_API_KEY=...`, and a new output directory:

```sh
python tools/compare_factorio_reviews.py \
  --trace report/factorio-shift-20260917/trace.json \
  --catalog /absolute/path/to/selected-models.json \
  --key-file /absolute/path/to/local-key-file \
  --output /absolute/path/to/new-output \
  --budget 0.50
```

Token use, provider routing, prices and timings can change on a rerun. The
reservation ceiling is checked before each request; missing charges retain the
full reservation. The table's dollar values use recorded charges, not a pricing
formula. Code and comparison tools are MIT licensed.