TexFaber

Journal / From the notebook

Decisions, Decisions

Testing fast, inexpensive decision models on fiction: the single-model results, the surprises about confidence and reliability, and why this young field makes me hopeful.

Jamie Heerlyn4 min read

A small robot considers a question-mark card over an open novel, beside test cards marked with ticks and crosses and an analogue stopwatch, in a textured cream-and-ink illustration.

I've spent the last couple of days testing decision models on fiction, and I've come away more hopeful about what they could make possible. These make a narrow judgement about a piece of text and return a confidence score. I wanted to know how often they would make a firm decision, and how often that decision would be wrong.

Those two numbers need to stay together. A model that leaves most things undecided can make very few mistakes. A model that decides almost everything can look useful until you count the answers it got wrong.

The latest full test covers 2,762 cases, taken from the first two chapters of The Riddle of the Sands, The Lodger, The Hound of the Baskervilles and The War of the Worlds. These are short readings from four books, rather than tests of the complete novels.

There were 18 models in that run. A follow-up through OpenRouter screened another 15 candidates, with Microsoft Decision-1.1 and Perplexity Decider V1.3 taken through the same complete test.

Here are some of the more useful individual results. Decided means the model gave a firm answer under the test's scoring rules; the remaining cases were left undecided. Wrong means the answer disagreed with the reference key. The error rate is calculated only over the cases the model decided.

Model Cases decided Share of all cases Wrong Error rate on decided cases
d1-ultra 1,605 58% 23 1.4%
Mammoth 1,906 69% 35 1.8%
Perplexity Decider V1.3 2,250 81% 66 2.9%
GLiDE 2,217 80% 70 3.2%
d1.2-ultra 1,948 71% 73 3.7%
Clef 1,195 43% 46 3.8%
Microsoft Decision-1.1 557 20% 23 4.1%
Snap 1,948 71% 86 4.4%
Jev 661 24% 39 5.9%
Drex Max 2,291 83% 155 6.8%

Scroll sideways to see all columns →

d1-ultra had the lowest error rate among the models that decided at least half the cases: 23 recorded mistakes in 1,605 decisions. Mammoth reached further, deciding roughly seven cases in ten, with 35 mistakes. Both were encouraging results.

Perplexity was the interesting addition from the follow-up. It decided 2,250 cases with 66 mistakes, giving it slightly more coverage and slightly fewer recorded errors than GLiDE. That is a close comparison on this material. Against Drex Max, it reached almost as many decisions with substantially fewer mistakes.

Microsoft Decision-1.1 was much more hesitant. It decided only about one case in five, yet its error rate on those decisions was higher than d1-ultra's, Mammoth's or Perplexity's.

What surprised me was that hesitation didn’t necessarily buy reliability. Some models left most cases undecided and still made proportionally more mistakes than models willing to answer much more. I’d expected a clearer relationship between caution and correctness.

There are limits to all of this. Each model had one run on the four extracts, and the reference answers were themselves assembled using AI. The key can contain mistakes, particularly where a passage allows more than one reasonable reading. These are rough development results, and small differences deserve caution.

What I found useful was seeing the trade-off in actual counts. Mammoth's 1.8% error rate still meant 35 wrong decisions, with 856 other cases left undecided. That gives me a much clearer picture of its performance than a headline accuracy figure would.

These are often called “System One” models, and the current wave still feels very young. TypeSafe introduced Jev on 15 September, only a couple of months ago. Jev seems to have helped bring a lot of attention to this approach; it heads the OpenRouter decision-model usage rankings available as I write.

The other thing that struck me was how quickly and cheaply I could get many of these judgements back. The additional OpenRouter experiment recorded about US$0.17 in API charges, helped by reusing existing answers and a free route for Perplexity. That excludes the earlier testing and the reference-answer work, but it made exploring a wider range of models feel very accessible.

I can see why this type of model would be useful as a building block in software. For a small project, speed and cost affect how much experimenting I can do and which ideas feel realistic to attempt. For TexFaber, I still need the judgements about a story to be dependable enough that a writer can use them.

If reliability improves while the speed and cost advantages hold, I'm hopeful that increasingly sophisticated AI-assisted systems will become attainable for many more projects. I hope we're still in the early days of what these models can achieve.

Jamie Heerlyn

Written from development test results with AI assistance.

Free early access

Get a free account

Join the waiting list for a free account when TexFaber is ready for early testing. Testing is still months away; there is no launch date yet. You’ll receive occasional progress updates. I’ll ask you to try it on real writing and share bugs and feedback.

I'll only use your email for TexFaber updates. Reply any time to be removed. How signups are handled.