ADMITBench: A Safety-Governed Reference Framework for Evaluating the Admissibility of Industrial LLM Advisories
This is the specification of the ADMIT framework. Everything else on this site, the gates, the verdict contract, the desks, is an implementation of what this paper defines.
In the authors' words.
This white paper presents ADMITBench, a reference framework for evaluating industrial LLM advisories at the level of the proposed action. The framework implements a versioned, safety-governed evaluation contract that checks whether a recommendation is supported by the available evidence, permitted under the stated authority and procedure, and acceptable under the plant-specific consequence checks encoded in the selected evaluation profile. In this report, safety-governed means that eligibility is determined through explicit, non-compensatory checks derived from a versioned plant profile; it does not mean that the evaluator, model, or plant has been safety-certified. Release 0.1.0 is a public reference implementation for technical and research evaluation, not an authorisation for physical execution.
The paper, in place.
Served straight from arXiv. If the viewer doesn't load in your browser, use the buttons above.
What it claims, in plain words.
Judge the act
The action record is the unit of evaluation, not the chat answer. What did the model propose to do, with what parameters, on what evidence, and when?
Gates before ranking
The checks are non-compensatory. A failed hard gate produces no score at all: the record is ineligible, not ranked low. No fluency elsewhere can buy it back.
Desks share the check
The same contract is applied at evaluation, in simulation, and at enforcement. One cartridge describes the plant; every desk reads it the same way.
What the repo implements.
Release 0.1.0 is a reference implementation of the framework for technical and research evaluation, not an authorisation for physical execution. Bench is where the paper's checks run: hard gates before any ranking, sealed oracles, replayable verdicts, a CLI.
Suggested citation.
“No score, tier verdict, simulation result, or benchmark comparison produced by ADMITBench should be interpreted as functional-safety certification, site acceptance, or permission to deploy an autonomous controller. Site-specific engineering review, management of change, functional-safety assessment, operator training, and accountable human approval remain necessary.”
Read the failures,
not the leaderboard.
ADMIT reports what leaked past which gate, under which conditions. Every released figure comes from the industrial reference profiles, with its run id shown. Not yet reported in arXiv v1; a versioned artifact is planned.
Models name the correct hazard far more often than they produce an admissible action. The gap is the point.
of all failed records in the 7-model release hit the consequence gate first: parameters too weak, or the action deferred too long.
unsafe picks in multiple-choice format (7 models × 420 items). In this evaluation the multiple-choice format did not expose the parameter-and-timing failures the action-record evaluation found.
Internal reference evaluation · 7 models · 25 cases · 343 action records · not yet reported in arXiv v1
Base-case admissibility, per model · extended 12-model run
“Admissibility spans 44%–80%; frontier models cluster 64–80%.” · run eval_2026-07-21 · temperature 0 · 5,628 evaluations
“P(inadmissible | diagnosis correct) = 33%” and “~96% of those failures land at the consequence gate T4”. The models diagnose the hazard, then about a third of the time propose an action the physics rejects: magnitude too weak, or deferral too slow. Source: run eval_2026-07-21. Note the denominators: the 57% headline counts all failed records in the 7-model release; the ~96% here counts only diagnosis-correct failures in this extended 12-model run. Both are real.
Simulator validity, from run eval_2026-07-21: CSTR convergence shows 0/15 verdict flips against a 16x finer integration, with a worst crossing-time deviation of 0.08%. Under ±10% parameter sensitivity, verdicts are stable to k0, thermal inertia, and rho_cp. But 7/15 boundary cases flip under ±10% on activation energy (ER), reaction enthalpy (dH), and heat-transfer coefficient (UA0). So boundary-case T4 verdicts carry a ±10%-parameter caveat, not plant-grade certainty.
What was run, exactly — the disclosure checklist for run eval_2026-07-21. If a results page cannot fill this table, treat its numbers as decoration.
| Item | Disclosure |
|---|---|
| Model | 12 models, named above · temperature 0.0 · max_tokens 4000 |
| Context | Base cases, caution ablations, and the multiple-choice floor |
| Execution footprint | 5,628 evaluations · $7.63 total API cost |
| Limitations | ±10% parameter sensitivity on ER, dH, UA0 · 7.3% run-to-run verdict flip rate, source not fully isolated · two cartridges, one author team |
“Only the admissible actions are scored. The rest are recorded and attributed by first-failing gate, never ranked.”
Seven plain words.
Under the tiers, the specification is seven plain words — record, authority, procedure, evidence, reversibility, checker, audit — plus a violation vocabulary of AAS codes, where one code names one failure at one gate. The safety knowledge is open-book, but the per-case answer key is sealed: oracle, acceptable_actions, load_bearing, diagnosis_accept, escalation_ok, time_to_hazard, and the case notes exist only for the gates and the scorer. The full contracts, the cartridge format, and the complete AAS catalog are specified in the paper above.