S1CogBench, a benchmark released on arXiv, evaluates seven open models and three API models on 24 established cognitive experiments, revealing that fast, structured "System One" decision models often fail to adapt to cognitive task demands like humans. The benchmark also demonstrates that high autonomous accuracy does not correlate with a model's ability to predict human behavior.
System One decision models, also referred to as typed decision models, return a single structured choice over a set of legal options in a single call. This typed interface allows software to execute responses directly for routing, guardrails, and automated gates. A small model of this kind can be invoked many times inside a larger system. However, a typed interface does not prove that the model uses the relevant evidence or adapts to feedback. S1CogBench maps these experiments to typed requests, scoring option probabilities, native choices, and response latency to isolate which cognitive demands the models handle and which they ignore.
Benchmark Structure and Scoring
The benchmark covers 24 tasks organized into ten domains, including evidence, categories, history, adaptation, exploration, information, preference, planning, social, and metacognition. Tasks such as the Wisconsin Card Sorting Test (WCST), Horizon, and Mouselab each have a human-prediction and an autonomous mode, yielding 27 evaluation modes. For each decision, an adapter provides the model with the visible state and the ordered legal responses. The model returns a probability distribution over those responses, which a readout rule then turns into a probability, a native choice, or an executed action. The authors record the distribution, the native choice, the executed action, the feedback, and the timing, keeping the model's distribution separate from randomness added at execution.
The benchmark reports nine shared measures, including accuracy, reward, pseudo-regret, mean absolute error (MAE), human negative log-likelihood (NLL), Brier score, latency at the 50th and 95th percentiles, and invalid-response rate. Human-response NLL scores the observed choice given its preceding history, while autonomous scores use executed trajectories to calculate total reward. The authors use participant or independent environment resampling for uncertainty, rather than trajectory steps. With 10,000 bootstrap resamples, the results show clear stability for top performers. Nox supports performance above uniform on 23 targets, while Intern-Decision does so on 21. Laya reliably beats uniform on only 7 targets, while CLM does so on 11.
For sequential human-response prediction, each model uses that participant's preceding history in the form supplied by the task adapter. Conditioning on this history does not update model weights. Histories can be complete, summarized, windowed, or absent, representing different information conditions. Responses can be categorical actions, ordinal ratings, numerical reports, or event probabilities. The fitting data and evaluation splits of cognitive references are documented separately, including cases without held-out participants.
Autonomous Accuracy vs. Human Prediction
The authors evaluated Laya, Intern-Decision-4B, Llama-3.1-8B-Instruct, Minitaur-8B, Qwen3.5-4B, Decision-2.0-Nox-4B, and CLM-v0.1-8B, alongside API models Jev, Haiku, and Claude5.5. Under original readouts, Nox leads the seven open models on 18 of 26 directional targets. Intern-Decision also reliably outperforms its Qwen3.5 backbone on 19 targets.
Illustrative image: axonite / Pixabay
Autonomous performance and human-response prediction rank models differently. In the WCST, Nox achieves the highest autonomous accuracy among open models at 32.1%, yet it predicts human choices worse than uniform choice (NLL 1.552 versus 1.386). Minitaur predicts human choices best in WCST (NLL 1.344), showing that acting well in a closed loop does not imply modeling human cognition. The authors note that Minitaur's human-prediction scores reflect in-distribution performance, as it was fine-tuned on human behavior from the Psych-101 dataset. When all models select their highest-probability legal option, Minitaur also achieves the highest accuracy at 34.3%. Sampling leaves CLM near uniform in both Horizon and WCST.
Overall accuracy can also mask a lack of sensitivity to changing task demands. In the reinforcement learning and working memory (RLWM) task, Nox achieves high accuracy but shows a much smaller memory-load effect than humans. Similarly, in the Horizon task, all eight evaluated models respond less than humans to increased future opportunities, with participant-paired intervals falling below zero. Learning curves also diverge sharply: Intern-Decision improves across repeated RLWM presentations, while Laya remains almost flat. In Mouselab, Qwen reveals 15.5 nodes per episode and earns the highest autonomous reward among open models at 17.40, close to the 17.55 obtained by revealing every node, yet it predicts human choices poorly.
In the Bayesian urn task, all evaluated models reliably beat uniform choice, with Nox achieving a mean absolute error of 0.175 compared to a uniform error of 0.349. In Horizon, Nox leads the seven open models in both human prediction (NLL 0.408) and autonomous reward (56.21), retaining the highest reward under shared argmax and sampling. The reference random walk (RW) policy, fitted to 25 training participants, predicts human choices slightly better (NLL 0.400) but earns less reward (54.56).
Latency and Interface Caveats
Latency tests within the benchmark show that generating long responses is significantly slower than direct decisions. However, matched one-token generation adds little overhead, with median request-level generation-to-forward latency ratios ranging from 1.0024 to 1.0202 across the tested models. API latency includes network and provider service time; Jev's median is 0.35 seconds over all 27 modes, against 2.60 to 3.59 seconds for Claude5.5 with adaptive thinking.
The benchmark results come with interface caveats. Laya's SDK rounds probabilities to four decimals, assigning zero probability to 192 of 108,248 human responses in self-assessment tasks and resulting in an infinite NLL score. Minitaur reports a near-constant 50% answer on 319 of 330 nonliteral-number items. Jev records infinite NLL in multiple-cue judgment, hybrid sampling, and politeness due to rounding its returned probabilities to two decimals. The authors note that training data and compute are not matched across the evaluated models, and results are reported by the authors on an unreviewed preprint.
Existing cognitive benchmarks evaluate language and multimodal models. CogGym evaluates 50 models on 258 single-turn experiments, and CogBattery compares 54 language models with human data on 13 cognitive paradigms. System1-Arena compares System One decision models through typed probability requests, closed-loop environments, latency, and cost, while TypedBench examines calibration, wording and option sensitivity, length, and asymmetric decision costs. S1CogBench extends these distinctions to explicit event probabilities, source-defined condition effects, and structured interfaces. It is the first benchmark to evaluate System One decision models on established cognitive experiments.
Sources
PNEUMETRON EDITORIAL
Pneumetron's pipeline pairs AI-assisted drafting with human editorial review before publishing — our goal is to make staying informed easier, not to replace real reporting.




