The Pneumetron
NEWSTOOLSINFRASTRUCTURE
Pneumetron.S1CogBench Benchmarks System One Decision Models on Cognitive Experiments
Share
Skip to article content
  1. Home
  2. ›
  3. News
  4. ›
  5. AI Research
  6. ›
  7. S1CogBench Benchmarks System One Decision Models on Cognitive Experiments
AI Research·November 22, 2026

S1CogBench Benchmarks System One Decision Models on Cognitive Experiments

BY PNEUMETRON|5 MIN READ · 1,035 WORDS5 MIN READ
Tools
Share

In This Article

  • Benchmark Structure and Scoring
  • Autonomous Accuracy vs. Human Prediction
  • Latency and Interface Caveats
  • Sources
Illustrative photo: cyberspace, data, wire, electronic
Illustrative image: jarmoluk / Pixabay

A new benchmark reveals that fast, structured "System One" decision models often fail to adapt to cognitive task demands like humans, and high autonomous accuracy does not guarantee human-like prediction.

Key Takeaways

  • 01Nox leads open models on 18 of 26 targets but predicts human WCST choices worse than uniform choice.
  • 02Minitaur predicts human choices best in WCST with an NLL of 1.344, proving autonomous success does not imply human-like cognition.
  • 03All eight models respond less than humans to increased future opportunities in Horizon, with intervals below zero.
  • 04Laya's probability rounding causes infinite NLL scores on 192 responses, and training data remains unmatched across models.

S1CogBench, a benchmark released on arXiv, evaluates seven open models and three API models on 24 established cognitive experiments, revealing that fast, structured "System One" decision models often fail to adapt to cognitive task demands like humans. The benchmark also demonstrates that high autonomous accuracy does not correlate with a model's ability to predict human behavior.

System One decision models, also referred to as typed decision models, return a single structured choice over a set of legal options in a single call. This typed interface allows software to execute responses directly for routing, guardrails, and automated gates. A small model of this kind can be invoked many times inside a larger system. However, a typed interface does not prove that the model uses the relevant evidence or adapts to feedback. S1CogBench maps these experiments to typed requests, scoring option probabilities, native choices, and response latency to isolate which cognitive demands the models handle and which they ignore.

Benchmark Structure and Scoring

The benchmark covers 24 tasks organized into ten domains, including evidence, categories, history, adaptation, exploration, information, preference, planning, social, and metacognition. Tasks such as the Wisconsin Card Sorting Test (WCST), Horizon, and Mouselab each have a human-prediction and an autonomous mode, yielding 27 evaluation modes. For each decision, an adapter provides the model with the visible state and the ordered legal responses. The model returns a probability distribution over those responses, which a readout rule then turns into a probability, a native choice, or an executed action. The authors record the distribution, the native choice, the executed action, the feedback, and the timing, keeping the model's distribution separate from randomness added at execution.

The benchmark reports nine shared measures, including accuracy, reward, pseudo-regret, mean absolute error (MAE), human negative log-likelihood (NLL), Brier score, latency at the 50th and 95th percentiles, and invalid-response rate. Human-response NLL scores the observed choice given its preceding history, while autonomous scores use executed trajectories to calculate total reward. The authors use participant or independent environment resampling for uncertainty, rather than trajectory steps. With 10,000 bootstrap resamples, the results show clear stability for top performers. Nox supports performance above uniform on 23 targets, while Intern-Decision does so on 21. Laya reliably beats uniform on only 7 targets, while CLM does so on 11.

For sequential human-response prediction, each model uses that participant's preceding history in the form supplied by the task adapter. Conditioning on this history does not update model weights. Histories can be complete, summarized, windowed, or absent, representing different information conditions. Responses can be categorical actions, ordinal ratings, numerical reports, or event probabilities. The fitting data and evaluation splits of cognitive references are documented separately, including cases without held-out participants.

Autonomous Accuracy vs. Human Prediction

The authors evaluated Laya, Intern-Decision-4B, Llama-3.1-8B-Instruct, Minitaur-8B, Qwen3.5-4B, Decision-2.0-Nox-4B, and CLM-v0.1-8B, alongside API models Jev, Haiku, and Claude5.5. Under original readouts, Nox leads the seven open models on 18 of 26 directional targets. Intern-Decision also reliably outperforms its Qwen3.5 backbone on 19 targets.

Illustrative photo: cyber, security, network, internet Illustrative image: axonite / Pixabay

Autonomous performance and human-response prediction rank models differently. In the WCST, Nox achieves the highest autonomous accuracy among open models at 32.1%, yet it predicts human choices worse than uniform choice (NLL 1.552 versus 1.386). Minitaur predicts human choices best in WCST (NLL 1.344), showing that acting well in a closed loop does not imply modeling human cognition. The authors note that Minitaur's human-prediction scores reflect in-distribution performance, as it was fine-tuned on human behavior from the Psych-101 dataset. When all models select their highest-probability legal option, Minitaur also achieves the highest accuracy at 34.3%. Sampling leaves CLM near uniform in both Horizon and WCST.

Overall accuracy can also mask a lack of sensitivity to changing task demands. In the reinforcement learning and working memory (RLWM) task, Nox achieves high accuracy but shows a much smaller memory-load effect than humans. Similarly, in the Horizon task, all eight evaluated models respond less than humans to increased future opportunities, with participant-paired intervals falling below zero. Learning curves also diverge sharply: Intern-Decision improves across repeated RLWM presentations, while Laya remains almost flat. In Mouselab, Qwen reveals 15.5 nodes per episode and earns the highest autonomous reward among open models at 17.40, close to the 17.55 obtained by revealing every node, yet it predicts human choices poorly.

In the Bayesian urn task, all evaluated models reliably beat uniform choice, with Nox achieving a mean absolute error of 0.175 compared to a uniform error of 0.349. In Horizon, Nox leads the seven open models in both human prediction (NLL 0.408) and autonomous reward (56.21), retaining the highest reward under shared argmax and sampling. The reference random walk (RW) policy, fitted to 25 training participants, predicts human choices slightly better (NLL 0.400) but earns less reward (54.56).

Latency and Interface Caveats

Latency tests within the benchmark show that generating long responses is significantly slower than direct decisions. However, matched one-token generation adds little overhead, with median request-level generation-to-forward latency ratios ranging from 1.0024 to 1.0202 across the tested models. API latency includes network and provider service time; Jev's median is 0.35 seconds over all 27 modes, against 2.60 to 3.59 seconds for Claude5.5 with adaptive thinking.

The benchmark results come with interface caveats. Laya's SDK rounds probabilities to four decimals, assigning zero probability to 192 of 108,248 human responses in self-assessment tasks and resulting in an infinite NLL score. Minitaur reports a near-constant 50% answer on 319 of 330 nonliteral-number items. Jev records infinite NLL in multiple-cue judgment, hybrid sampling, and politeness due to rounding its returned probabilities to two decimals. The authors note that training data and compute are not matched across the evaluated models, and results are reported by the authors on an unreviewed preprint.

Existing cognitive benchmarks evaluate language and multimodal models. CogGym evaluates 50 models on 258 single-turn experiments, and CogBattery compares 54 language models with human data on 13 cognitive paradigms. System1-Arena compares System One decision models through typed probability requests, closed-loop environments, latency, and cost, while TypedBench examines calibration, wording and option sensitivity, length, and asymmetric decision costs. S1CogBench extends these distinctions to explicit event probabilities, source-defined condition effects, and structured interfaces. It is the first benchmark to evaluate System One decision models on established cognitive experiments.

Sources

  • arXiv: S1CogBench: What System One Decision Models Can and Cannot Do on Cognitive Experiments
#airesearch#benchmarks#decisionmodels#cognitiveai#llmevaluation
PE
PUBLISHED BY

PNEUMETRON EDITORIAL

Pneumetron's pipeline pairs AI-assisted drafting with human editorial review before publishing — our goal is to make staying informed easier, not to replace real reporting.

Source Material:arXiv ↗
Share this article
Share
Stay Informed

Never miss a signal.

Subscribe to the Pneumetron Intelligence Digest.

← Previous
Lang-SVG introduces hierarchical trees for semantic image vectorization
Next →
Probing and steering directions diverge in modern language models

More from AI Research

View All →
AI Research9 min ago
Illustrative photo: switch, it, server, server cabinet

Probing and steering directions diverge in modern language models

An arXiv paper shows the best direction to detect a concept in a model’s hidden state is rarely the best one to steer it. Transfer laws also differ, and steering toward abstention works best on uncertain errors.

BY PNEUMETRON5 MIN READ
Read more
AI Research12h ago
Illustrative photo: laptop, macbook, codes, coding

Lang-SVG introduces hierarchical trees for semantic image vectorization

Researchers have published Lang-SVG, a method that reconstructs raster images as hierarchical, semantically labeled SVG trees rather than flat layers. The approach uses open-vocabulary segmentation and MLLMs to organize vector primitives into parent-child structures, improving both structural alignment and rendering fidelity under strict control-point budgets.

BY PNEUMETRON5 MIN READ
Read more
AI Research12h ago
Illustrative photo: server, cloud, development, business

AtomicChat Releases Uncensored LoRA for Inkling-2-Mini

AtomicChat published a rank-1 LoRA adapter that strips refusal behavior from the 26.4B MoE Inkling-2-Mini model. The edit drops harmful refusal rates to zero while costing 1.5 points on MMLU and spiking first-token KL on previously refused prompts.

BY PNEUMETRON4 MIN READ
Read more
AI Research12h ago
Illustrative photo: network, server, system, infrastructure

Open-source guard scans iOS and Android projects before app store submission

A new open-source playbook and shell-based guard automates pre-submission checks for mobile apps, integrating with AI coding agents to block submissions against 117 known rejection patterns and 95 regulatory deadlines.

BY PNEUMETRON4 MIN READ
Read more

Comments

In This Article

  • Benchmark Structure and Scoring
  • Autonomous Accuracy vs. Human Prediction
  • Latency and Interface Caveats
  • Sources

Most Read

01
AI Research·3d ago
Tencent releases LoopForge to orchestrate coding agents in self-hosted pipelines
02
AI Research·5d ago
Four New AI Papers Advance Reasoning, TTS, and Decoding Efficiency
03
AI Research·5d ago
Recent AI Research Reveals Gaps in Coding Agents and Advances Scientific Modeling
04
AI Research·5d ago
Anthropic Upgrades Fable and Mythos as Frontier AI Expands Open-Source and Robotics
05
AI Research·3d ago
Game2Player Scales Visual Agent Training with Co-Developed Game Environments
Daily Digest

Get top AI & tech signals delivered to your inbox every morning.

Subscribe →
Follow Signals
X / TWITTERLINKEDININSTAGRAMYOUTUBETELEGRAM

THE PNEUMETRON

The publication of modern technology, AI, and infrastructure.

AI-assisted, human-reviewed.

NEWS CATEGORIES

  • AI RESEARCH
  • TECHNOLOGY
  • WORLD
  • BUSINESS
  • POLITICS
  • SCIENCE
  • HEALTH
  • SPORTS
  • ENTERTAINMENT

AI & RESEARCH

  • AI RESEARCH
  • INFRASTRUCTURE
  • TOOLS
  • DEEP DIVES
  • TECHNICAL DISPATCHES

RESOURCES

  • ABOUT US
  • CONTACT
  • CAREERS
  • ADVERTISE
Pneumetron.
© 2026 Pneumetron. AI-assisted, human-reviewed.
All rights reserved.
ABOUTTOOLSPRIVACYTERMSCONTACT