👉 GnollBench, the Gnoll Overseer's AI benchmarking system, measures how well the Overseer works with a given model, while public benchmarks such as those of Artificial Analysis measure what a model can do in general.
📖 Two Different Questions
All AI benchmarks give models a fixed set of tasks and turn the results into numbers. Where they differ is the question the numbers answer.
| Public benchmarks | GnollBench | |
|---|---|---|
| The question | "How capable is this model in general?" | "How good is the Gnoll Overseer when this model drives it?" |
| What is tested | The model, under neutral conditions | One product, with its own instructions and lookup tools |
| Subject | Mathematics, science, coding, office work, and more | GnollHack only |
| Made for | Everyone who has to choose a model | The Overseer's developers |
A public index cannot tell you whether a model gives sound advice about fighting a cockatrice, and GnollBench cannot tell you whether a model is any good at law or physics. Neither question can be answered with the other's tool, so GnollBench complements the public benchmarks rather than competing with them. Introduction to GnollBench explains GnollBench itself for newcomers.
🌍 The Public Benchmarks in Brief
There are hundreds of AI benchmarks. The ones below are well known, and each tests in a different way.
| Benchmark | Who decides what is "good" | In one sentence |
|---|---|---|
| Artificial Analysis | Program tests and AI judges | Ten different tests rolled into one index, plus live speed and price tracking |
| Arena | Human voters | People pick the better of two anonymous answers |
| Expert exams | An answer key with exact answers | Very hard questions written by specialists |
| LiveBench | An answer key with exact answers | Questions that are replaced regularly, so models cannot memorize them |
| HELM | Many different metrics | The broadest and most transparent academic test collection |
| HealthBench | An AI grader with physician-written answer keys | Health conversations graded against detailed checklists |
| BALROG | The game itself | AI models play games, including NetHack |
📈 Artificial Analysis
Artificial Analysis is an independent company that tests AI models, and the services that host them, and publishes the results. Its headline number is the Intelligence Index (version 4.3.2 at the time of writing), which combines ten separate tests in four groups:
| Group | Share of the index | What the model has to do |
|---|---|---|
| Agents | 30% | Carry out long, multi-step work tasks by itself |
| General | 30% | Recall knowledge, reason over long documents |
| Coding | 20% | Write programs that pass tests |
| Scientific reasoning | 20% | Solve very hard science problems |
The ten tests range from 66 long tasks to 6,000 short questions. Code is graded by program tests, short answers by an AI "answer checker", and open-ended work by panels of several AI judges. From repeated runs, Artificial Analysis estimates that the index is accurate to within ±1%. Speed and price are tracked separately and continuously: each hosted model is called eight times a day, and the site shows the typical result of the last 72 hours.
One of the ten tests subtracts points for a wrong answer, while "I don't know" costs nothing, so a model that bluffs scores worse than one that admits ignorance. GnollBench's critical error rule comes from the same worry, and GnollBench now treats honesty the same way: an honest, specific "I could not verify this" scores at least 50 out of 100, about as much as a substantially wrong answer and at least twice as much as a dangerously wrong one.
🗳️ Arena
Arena started in 2023 as Chatbot Arena at the University of California, Berkeley, was later called LMArena, and has carried its current name since January 2026. It has no answer key. A visitor asks anything they like, two anonymous models answer, and the visitor votes for the better answer before learning which models they were. Millions of such votes are turned into ratings, much like chess ratings.
Arena measures what people prefer, which is hard to measure any other way, but a preferred answer is not always a correct one: a confident, nicely formatted answer can win the vote while being wrong.
🎓 Expert Exams
These are very hard questions written by specialists, with answers that can be checked exactly. Well-known ones are Humanity's Last Exam, GPQA Diamond, and Epoch AI's FrontierMath.
FrontierMath has 338 problems, and only twelve of them are public, so models cannot have seen the rest during training. Epoch AI runs most models 16 times on GPQA Diamond to average out luck.
Specialists make mistakes too. In June 2026, Epoch AI released a major FrontierMath update that fixed errors in 42% of the problems. GnollBench also has routines for checking its answer keys.
🔬 LiveBench and HELM
- LiveBench fights memorization by regularly replacing its questions with new ones drawn from recent material. Answers are checked against exact solutions, with no AI judge involved.
- HELM, from Stanford University, takes the broadest view: many situations measured in many ways, not only accuracy. It is also transparent, publishing every question and every model reply.
🩺 HealthBench
HealthBench, published by OpenAI, is the benchmark closest to GnollBench in method. It consists of 5,000 health conversations, and 262 physicians wrote a separate checklist for each one: 48,562 criteria in all, each worth a number of points. An AI grader goes through every checklist.
HealthBench's authors tested whether their AI grader could be trusted, and found that it agreed with physicians about as often as physicians agree with each other. GnollBench grades the same way on a far smaller scale, but has no such study behind it, and its checklists are usually drafted by an AI and reviewed by the developers, not written by outside experts.
🕹️ BALROG
BALROG is the closest in subject: it makes AI models play games, the hardest of them NetHack, and scores how far they get.
- In the original 2024 study, the best models averaged under 2% of the way through NetHack.
- A blog report from January 2026 described a best run that reached dungeon level 10, still only about an eighth of the way by BALROG's measure.
- In September 2026, the same author reported that an AI agent had ascended, winning the game outright, though outside BALROG's own test setup.
GnollBench does not ask a model to play the game, but to advise a human who plays it.
📊 Side-by-Side Comparison
🎯 What Is Tested
| GnollBench | Artificial Analysis index | Arena | Expert exams | |
|---|---|---|---|---|
| Main purpose | Improve one product | Compare models for everyone | Rank models by human preference | Track the limits of AI ability |
| What is tested | The model inside the real Overseer chat | The model under neutral conditions | The model as a chat partner | The model by itself |
| Subject | GnollHack | Agents, coding, knowledge, science | Whatever visitors ask | Science, mathematics, knowledge |
| Size | 18 questions in the default suite; custom suites of any size up to 50 | Thousands of items | Millions of votes | Hundreds to thousands of questions |
| Who writes the questions | Usually an AI that can consult the game's source code and this wiki; the developers review them | Artificial Analysis and outside authors | The visitors | Subject specialists |
| Who writes the answer keys | Usually an AI, from the same sources; improved between runs in the light of the results | The authors of each test | There are none | Subject specialists |
| Tools for the model | 16 of the Overseer's 31 tools: the lookups for the wikis, game data, and source code | Code and web tools in the agent tests | Usually none | Usually none |
The Overseer's other tools are left out of the exam because they do not suit a fixed test: they read the player's own game and files, search GitHub or the server's game records, or hand work to helper AIs. When a suite is built around a game situation, that situation is normally captured from a real game with the game's own snapshot export.
📏 How It Is Measured
| GnollBench | Artificial Analysis index | Arena | Expert exams | |
|---|---|---|---|---|
| Who grades | Two AI graders from different companies, with a written answer key | Program tests, AI checkers, AI judge panels | Humans | An exact answer key |
| Score | Index up to 100; hard questions count more | Weighted average of ten tests | Chess-like rating | Percentage correct |
| Handling of chance | Uncertainty range on every run; repeated runs | Repeated runs; stated ±1% | Uncertainty range on every rating | Repeated runs by some publishers |
| Speed | The model's own thinking time per answer | Live speed of each hosting service | Not measured | Not measured |
| Cost | Estimated cost of each run, per role | Price lists, and the cost of running the index | Not measured | Usually not measured |
| Results | Preliminary results published as an executive summary; run details internal, available on request | Public | Public | Public |
An AI judge, a model that grades another model's answers, is fast and cheap, but it has blind spots of its own, which is why benchmarks that use one surround it with checks.
✨ What Is Unique About GnollBench
Few of these ideas are unique on their own, and the last column lists related ideas elsewhere. What is unusual is the combination, and that it serves one purpose: making one product better.
| Feature | What it means | A related idea elsewhere |
|---|---|---|
| It tests the real product | The model gets the very same instructions as the live Overseer chat, and the same lookup tools for the wikis, game data, and source code | Public benchmarks must use neutral conditions to be fair to every model |
| Finding faults comes first | The main goal is to trace wrong answers to a fixable cause. Comparing models comes last. | Most benchmarks exist to rank models |
| A rule for dangerous advice | A confident falsehood that would hurt the player caps the answer at 25 out of 100. The answer key must show it to be false, and the grader must quote the sentence word for word. | Artificial Analysis's penalty for bluffing; HealthBench's criteria with negative points |
| Honesty is not punished | An honest "I could not verify this", with nothing false in it, scores at least 50, more than a dangerous guess | Artificial Analysis's penalty for bluffing; SimpleQA's "not attempted" grade |
| Two graders score; the rest check | Two AI graders from different companies grade every answer, and the score is their average. A third reader and a fact-checker examine answers but do not set scores, so results stay repeatable. | Juries of AI judges from different companies; Arena's voters also judge blind, without knowing which model wrote what |
| Answers are checked against the game | A fact-checking AI looks claims up in the game's wiki and source code. Knowing more than the answer key is not punished. | Rare, because few benchmarks cover a single piece of software |
| No misleading numbers | If the AI company's service failed during a run, no indices are shown at all | Reporting uncertainty ranges is common; withholding results is not |
| Real game situations | Questions can be tied to a situation captured from a live game: "what should I do here?" | BALROG also puts models into real game states, as players |
| The exam examines itself | Questions that everyone passes, or whose scores jump around, are flagged for a human, and answer keys are improved between runs | Expert exams retire or repair questions too, as FrontierMath did |
💡 Example: Suppose a model answers a question about a wand wrongly. For a ranking benchmark, that is one lost point. For GnollBench it is a lead: was the fact missing from this wiki, or did a lookup tool return something confusing? Once the cause is fixed, every model in the Overseer answers better, so a failed question is often worth more to the developers than a passed one.
💪 Where the Public Benchmarks Are Stronger
GnollBench is a small, internal tool, and in these respects the public benchmarks are better:
| Strength | Public benchmarks | GnollBench |
|---|---|---|
| Scale | Thousands of questions, so chance evens out | 18 questions in the default suite and 50 at most in a custom one, so chance plays a large part |
| Breadth | Many fields, long documents, images, long conversations, multi-step work | Single questions about one game |
| Grading | Program tests, exact answers, and human votes do not share AI graders' blind spots | Two AI graders from two companies set every score |
| Openness | Results can be checked and criticized by anyone | Only an executive summary of preliminary results is public. Run details remain internal, so the method must largely be taken on trust. |
| Independence | The testers do not build the products they test | Reviewed and run by the team that builds the Overseer |
| Live speed data | Measured around the clock, across hosting services | Only seen during its own runs |
Repeating a short exam many times shows how steady a model is, but it never makes the exam any bigger: the uncertainty that comes from having few questions can only be reduced by adding questions. GnollBench allows custom suites larger than the default 18, but even the largest stays far below the size of a public benchmark.
🚧 Limitations on Both Sides
🐺 GnollBench
| Limitation | Why it matters |
|---|---|
| Small suites | With 18 questions in the default suite, and 50 at most in a custom one, a few lucky or unlucky answers move the result |
| AI-drafted questions and answer keys | Human review catches mistakes, but an AI's blind spots can still slip into the exam |
| Answer keys keep changing | Improving an answer key makes the exam better, but results from before and after the change can no longer be averaged together |
| AI graders | Graders from two companies balance each other, but every score still rests on AI judgment, however carefully it is constrained. Changing a grader starts a new series of results. |
| AI-rated difficulty | The ratings that make hard questions count more are themselves made by an AI |
| Mostly internal | Only a summary of preliminary results is public. Run details can be requested, possibly anonymized, but outsiders cannot check them independently |
| Single questions only | Longer conversations, spoiler-free mode, and web search are not measured |
| One subject | A good result says nothing about anything other than GnollHack |
🌍 Public Benchmarks
| Limitation | Why it matters |
|---|---|
| Contamination | Published questions end up in training data, and scores rise without models getting smarter |
| Saturation | Once the best models score near the top, the benchmark stops telling them apart |
| Neutral conditions | A product's own instructions and tools can change a model's behavior a great deal |
| Preference is not correctness | In vote-based rankings, a confident wrong answer can beat a careful right one |
| An average hides the details | A high overall index can conceal a weak spot in exactly the area you need |
| Versions move | When the contents of an index change, old and new scores cannot be compared directly |
Contamination means that a benchmark's questions and answers have leaked into the material a model was trained on. The model then "remembers" answers instead of working them out, like a student who saw the exam paper in advance. Saturation means that a benchmark has become too easy for the best models: if everyone scores 98%, the exam no longer shows who is better.
GnollBench has the version problem too. It records the exact conditions of every run and refuses to average runs whose conditions differ.
🤝 How They Complement Each Other
In practice, the two kinds of benchmark are used one after the other:
- Public benchmarks make the shortlist. Nobody can test every model on everything, and public figures on ability, speed, and price show which models deserve a closer look.
- GnollBench tests the shortlist inside the real product, showing how each candidate behaves with the Overseer's instructions and tools on GnollHack questions.
- Wrong answers lead to fixes in this wiki, in the tools, and in the instructions, which helps whichever model a player picks.
Either kind alone misses something:
| If you only looked at… | You would miss that… |
|---|---|
| Public benchmarks | A top-ranked model can still do poorly in a narrow subject where it has to look things up with unfamiliar tools |
| GnollBench | A model that does well has shown that it is a good GnollHack advisor, and nothing more |
The models this process selected for the Overseer are listed in Choosing AI Model for Gnoll Overseer.
🎲 Interesting Facts
- The NetHack Learning Environment was released for AI research in 2020. In a 2021 competition built on it, hand-written bots beat the machine-learning entries by a wide margin.
- A hand-programmed bot called BotHack completed NetHack in 2015. In September 2026, an agent driven by OpenAI's GPT-6 Astra made what its operator believes is the first recorded ascension by an AI language model: a dwarven Valkyrie, on its third run, after 37,140 turns. It had free access to spoilers, the wiki, and the game's source code, and built its own tools along the way. A human still helped with setup and suggestions, and stopped and resumed its sessions.
- GnollBench's rating scale, the Behaviorally Anchored Rating Scale, was developed by psychologists in 1963 for judging job performance and was first used on nurses, long before anyone graded an AI with it.
- With 6,000 questions, a single component of the Artificial Analysis index is over 300 times the size of GnollBench's default suite, and 120 times the largest suite GnollBench allows.
- Artificial Analysis measures how fast a hosting service delivers text. GnollBench measures how long the model itself spends on a whole answer, with the waiting time for lookups removed.
- GnollBench is an open-book exam: the model may search the wiki during the test, just as it does in the live chat. Having memorized the material matters less than finding and using it correctly, which also makes contamination less of a worry.
- Artificial Analysis and GnollBench both settled on the same three-way view of a model: quality, speed, and cost.
📖 Learn More
- GnollBench — All GnollBench guides in one place.
🔗 External Links
- Artificial Analysis: Intelligence Benchmarking Methodology
- Artificial Analysis: Performance Benchmarking Methodology
- Arena FAQ
- Epoch AI Benchmarks
- LiveBench
- HELM
- HealthBench
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
- An LLM Beat NetHack
📅 Validity
Public benchmarks as of September 2026. GnollBench as of harness version 45 and scoring method version 13 (October 2026).