Paleo Bench

55 model configurations, 7 Byzantine Greek folios

Run 24 October 2026

Graphs

  • One dot per model, coloured by provider; open weight models share a colour
  • Higher: better score; further left: cheaper or faster
  • Select a dot to see that model below

Score against cost

0255075100$0.01$0.10$1.00Mean score, higher is better← cheaperTotal cost, log scalepricier →Gemini 4 Argon (multi-step)Gemini 4 Argon (low)Gemini 3 Flash (low)Gemini 3.5 Flash (low)Haiku 5.5 (low)DeepSeek V4.1 Flash (no thinking)Llama 4 Scout

Score against time

02550751001 min3 min10 min30 min1 h2 hMean score, higher is better← fasterTotal time, log scaleslower →Gemini 4 Argon (multi-step)Gemini 4 Argon (low)Gemini 3 Flash (low)
  • Failed at least one folio

Not plotted: Gemini 3.5 Flash (high) (no cost recorded; time graph only)

Gemini 4 Argon (multi-step)

Google, low-thinking draft, medium-thinking checks, position 1

Mean score
98.1
Colour images
98
Microfilm
99
Folios run
7 of 7
Cost
$0.28 in total, $0.040 a folio
Time
4.5 minutes in total, 38 seconds a folio
Open image & transcription

Quick answers

  • Best score: , 98.1, $0.28, 4.5 min
  • Within a point of the best: , 97.3 for $0.062, 4.5× cheaper and 2.4× faster
  • Best from Anthropic: , 94.6, $0.74
  • Best from OpenAI: , 78.2, $0.75
  • Best open weight model: , 70.9, $0.14
  • High thinking beats low in 4 of 13 models, at a median 4.2× the cost
  • Best on microfilm: , 99
  • Score: 100 minus character error rate, rounded down, minimum 0
  • Scoring ignores accents, punctuation and capitals
  • Every sigma form counts as σ; ϑ, ϐ and similar variants as their usual letters
  • ϛ, ȣ and ϗ count as στ, ου and και
  • Latin or Cyrillic look-alikes inside Greek words count as Greek
  • Means skip failed folios
  • Ground truth keeps rubrics on GA L60 and GA L587 but omits a faded one on GA L2010
  • Time: summed request time, timeouts included
  • Cost: US dollars; OpenRouter figures estimate from listed token prices