LLMs Struggle with Statistical Self-Consistency, Macro Fallacy Identified

What Changed

A recent study titled "Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models" investigates the extent to which Large Language Models (LLMs) adhere to fundamental probabilistic identities, specifically the law of total probability. This law dictates that prior-weighted conditional distributions should aggregate into population-level marginals when considering any valid partition of a population. The research uncovers widespread violations of this self-consistency principle across various problem domains and state-of-the-art frontier models.

The core methodology involved using binary trees to recursively partition a population into increasingly granular subpopulations. LLMs were then prompted with verbalized descriptions of these subpopulations, and their estimates were aggregated back to population-level estimates. These aggregated results were then compared across partitions of differing granularity. The consistent observation of inconsistencies suggests that while LLMs may possess relevant subpopulation knowledge, they do not reliably propagate this information into accurate aggregate estimates.

Crucially, the study identifies a phenomenon termed the "macro fallacy." This fallacy describes a pattern where estimates reconstructed from more fine-grained subpopulation responses are often better aligned with human reference data than direct population-level estimates. This effect was found to persist across variations in tree structure and estimation tasks, and could be partially mitigated through implicit prompting.

Technical Details

The research frames in-context learning as a form of conditional inference, where a prompt defines a context, and the model's output is an estimate of the corresponding conditional distribution. Under this interpretation, LLM estimates should naturally satisfy basic probabilistic identities, such as the law of total probability. This law is central to the study's evaluation framework.

To test this, the researchers constructed an evaluation scaffold using binary trees. This hierarchical structure allowed for the systematic partitioning of a population into progressively finer-grained subpopulations. For instance, a population could be initially divided into two broad groups, which are then each further divided, and so on, creating a tree of subpopulation descriptions.

LLMs were then presented with prompts that included these verbalized subpopulation descriptions. The models generated estimates for each subpopulation. These individual subpopulation estimates were then weighted by their respective prior probabilities (derived from the partition structure) and aggregated to form an overall population-level estimate. This aggregated estimate was then compared against a direct population-level estimate obtained by prompting the LLM without any subpopulation partitioning, as well as against human reference data.

Violations of statistical self-consistency were observed when the aggregated estimates from partitioned data did not match the direct population-level estimates or when both deviated significantly from human reference data. The macro fallacy specifically highlights that the aggregated estimates from more granular partitions often outperformed the direct population-level estimates in terms of accuracy against human benchmarks. This suggests that the act of breaking down a complex problem into smaller, more specific sub-problems, even if the LLM struggles with perfect aggregation, can sometimes lead to a more accurate overall understanding than a single, broad query.

Implicit prompting was found to partially recover some consistency, indicating that the way subpopulation descriptions are presented to the model can influence its ability to integrate information. This suggests that the issue might not solely be a lack of knowledge, but also a challenge in how that knowledge is accessed and utilized under different prompting strategies.

Developer Implications

For developers working with LLMs, these findings have significant implications for how models are prompted and how their outputs are interpreted, especially in tasks requiring statistical aggregation or inference over diverse populations. Relying solely on direct population-level queries for complex statistical estimations may lead to less accurate results than expected.

Developers might need to adopt more sophisticated prompting strategies that involve explicitly partitioning a problem space and aggregating results. This could involve:

• Decomposition: Breaking down complex estimation tasks into smaller, more manageable sub-tasks, prompting the LLM for each sub-task, and then programmatically aggregating the results. This mirrors the 'Partition, Prompt, Aggregate' methodology.
• Careful Prompt Engineering: Experimenting with different ways to describe subpopulations and the overall population to the LLM. The observation that implicit prompting can partially recover consistency suggests that subtle changes in prompt wording can have a measurable impact.
• Validation of Aggregated Outputs: Implementing robust validation steps to cross-check LLM-generated aggregate statistics against known probabilistic identities or human reference data, especially when dealing with partitioned data.
• Awareness of the Macro Fallacy: Recognizing that prompting for fine-grained details and then aggregating might yield more reliable results than a single, high-level query, even if the aggregation process itself is imperfect within the LLM.

These insights suggest a need for more structured and systematic approaches to leveraging LLMs for quantitative analysis, moving beyond simple, direct queries for aggregate statistics.

Bottom Line

The study "Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models" reveals a fundamental limitation in current LLMs: their struggle with statistical self-consistency. Despite their impressive capabilities in in-context learning, LLMs frequently fail to adhere to basic probabilistic identities, particularly the law of total probability when aggregating information across partitioned data. The identification of the "macro fallacy" underscores this, demonstrating that breaking down a problem into finer-grained subpopulations and then aggregating the LLM's responses can often lead to more accurate overall estimates than direct population-level queries. This suggests that while LLMs may possess the underlying knowledge, they face challenges in reliably propagating this knowledge into coherent aggregate statistics. For developers, this necessitates a more deliberate and structured approach to prompting and result aggregation, emphasizing decomposition and careful prompt engineering to mitigate these consistency issues and improve the reliability of LLM-based statistical inferences.