The Method

The Pipeline

Five stages per item, in one direction.

A creative-writing benchmark must defeat memorisation: asking a model to continue a famous novel mostly tests whether it has read the novel. NextChapterBench therefore never asks for the chapter that exists; every brief demands a chapter that was never written.

Divergent Briefs

Each brief departs from the published plot on purpose.

Where Austen sent the Elliots to Bath, the brief may forbid Bath outright. The required chapter exists nowhere in training data, so a model that merely remembers the book is remembering the wrong book. Each brief specifies required plot beats, a set of prohibitions, a target end-state, and a word range. The chapter is graded against the assignment, in the manner of a commissioned novelist.

Continuity Traps

Hidden facts per item, checked against the source chapter.

Each brief plants five small, checkable details carried over from the source chapter: a name, an hour, a distance, the placement of a door. They are never announced, and contradicting one counts double in the compliance score. Run ‘first’ has already caught such slips: a cylinder landed on the wrong evening in the Wells item, and a door attributed to the wrong character in the Burnett item; both were cited by the judges with chapter and verse.

The Judge

One judge, one call per chapter, the same company every time.

Every chapter is marked by a single judge, Sonnet 5, in one call that always looks the same: the source chapter, the brief, two continuations of the brief in shuffled order, and the author’s real next chapter, declared as the author’s. One of the two continuations is the chapter under test; the other is a fixed reference, the chapter Fable 5.1 (xhigh) wrote for that item in September 2026, frozen in the repository so that a model judged next year is judged in exactly the company of one judged today. The judge is not told which of the two is the candidate.

It grades all three chapters on eight dimensions of craft, 0–100: voice, prose, character, scene, integration, momentum, restraint and specificity, with no anchors beyond the ends of the scale (100 is as good as the finest published fiction in this voice could be, 50 is competent commercial prose). It checks the two brief-followers for compliance: every required beat, every prohibition, the end state and the hidden continuity traps, each pass or fail backed by a verbatim quotation, or it is a fail. And for every chapter it writes one plain sentence naming the worst flaw and one naming the best quality. The reference chapter’s scores are read in every call, which makes it the most-measured text on the board and the benchmark’s built-in noise gauge; it is not a contestant and does not appear in the standings. The reference model’s own entry is a fresh draft of the same ten chapters, judged beside its frozen draft like any other candidate. The author’s chapter is read on the five dimensions that are about the writing (voice, prose, character, restraint, specificity), not on scene, integration or momentum, because it was written to the real book and not to our brief.

Why one judge, and why this one. The benchmark began with a panel of three judges from three houses marking craft on a five-point scale, and by September 2026 that scale had run out of room: the top five models were receiving a 4 on eight marks in ten and the panel could not separate them. Rather than re-judge with three models, we auditioned candidates on the same chapters. Sonnet 5 agreed most closely with the other two candidates (rank correlation 0.84 with GPT-5.6 Sol, 0.73 with Gemini 3.1 Pro, against 0.75 between those two); it placed the real author ahead of every model on both audition items, where Gemini put Wilkie Collins at 74 and a model at 99; it used the middle of the scale where Sol scored the authors at 98 and 99 and left no room above; and, relative to the other two, it was 7.5 points harsher on Anthropic chapters, where Sol favoured OpenAI chapters by 11.8. All three put the same houses at the top. A single judge is a deliberate simplification, and its house is disclosed: it is an Anthropic model, and Anthropic models lead the table.

The Memorisation Check

A mechanical screen beneath the judges.

A mechanical layer screens every output for refusals, preambles, truncation, stray markdown, and, centrally, n-grams copied verbatim from the supplied chapter or from the real published continuation. In the pilot it flagged a 30-consecutive-word run of Austen’s actual Chapter II inside the top model’s output: passages the judges had scored highly turned out to be borrowed, and both facts now appear in the record. Aesthetic judgment and mechanical screening are kept deliberately separate.

Scoring

One formula, and one worked example from run ‘first’.

Effort, Noise, and What a Rank Means

How the settings were chosen, and how far apart two scores must be to mean anything.

Every model is run at an explicit reasoning-effort setting, named in the sub-line under its name. Where a vendor offers several, the standings show the best-scoring one and list the others beneath it; Show every setting expands them into rows of their own. The Anthropic models were first run at their command-line default, which is high; the OpenAI models at medium, Codex’s default; Gemini at the effort baked into its model name.

Effort is not a neutral dial, and it is not the same dial for every house. Under the earlier three-judge panel, Fable 5.1 gained a clear step only at xhigh, at about two and a half times the cost of high, while Opus 5, GPT-5.6 Sol, GPT-5.5, GPT-6 Astra and both Gemini Flash generations gained nothing measurable from any setting: their higher-effort runs sat inside the noise of their lower ones, and their reasoning budgets barely moved. Effort settings are therefore shown as variants of one model, never as separate contestants, so that a vendor cannot fill the table by running the same model five ways.

The noise gauge is built in. The reference chapter is scored in every call, so each item yields as many readings of the same text as there are models on the board, and the spread of those readings is the judge’s per-call noise on that item. The ± under each headline on the standings is the standard error of a model’s ten per-item scores. Two rows whose bands overlap should be read as a tie. Measured across the board, the reference chapter moves by about 3 points between calls, so a ten-item craft mean carries roughly ±1. Generation noise is larger and was measured directly on the leader: Fable 5.1 at xhigh wrote its ten chapters twice, and the second draft, judged beside the first in the same calls, came out 1.3 craft points lower on average, ahead on three items of nine, with single chapters moving by up to 7 points either way. Two models within about 2 points of each other on the headline are therefore not separable; the top of the table should be read as a tier.

A setting marked ✳ could not complete every item and is shown for reference only; a row marked — has not been judged yet. In the current run that is Fable 5.1 at low and Fable 5 at medium, both of which are stopped on The Secret Garden by the model’s own output content filter, on every one of twelve attempts, at those settings only. The chapter is the most memorised text in the corpus, and the likeliest explanation is that at low effort the model reproduces Burnett closely enough to trip a filter against recitation. A refusal that is returned as text, by contrast, is judged like any other chapter and scores zero on compliance, which is how GPT-5.6 Terra’s refusal to write in Hemingway’s voice was scored. Output that is not a chapter at all is treated like a timeout and regenerated: once in this run, Fable 5.1 at xhigh returned ten thousand words of tool-call markup with the chapter buried inside it and a loop of shell commands after; the replacement was judged. Two models needed more room than the default output budget: GLM 5.3 Flash and Hy3 spend most of their budget thinking before they write, and three of GLM’s chapters were cut off mid-sentence until the budget was doubled.