WearableQA: Benchmarking LLM Reasoning on Longitudinal Health Data

What Changed

The integration of Large Language Models (LLMs) into healthcare diagnostics has largely focused on static clinical notes or textbook medical knowledge. However, the real-world application of AI in digital health requires the ability to parse continuous, noisy, and longitudinal data streams—the kind generated by modern wearable devices. The introduction of WearableQA marks a shift in this evaluation paradigm. This benchmark moves away from static medical QA and forces models to grapple with the messiness of real-world physiological data.

Researchers have released a dataset comprising 4,084 multiple-choice questions derived from the actual health records of 200 individuals. Each record spans up to 500 days of daily measurements, including wearable time series, blood biomarkers, and demographic information. By preserving the noise, inter-individual variability, and long-term trends inherent in human physiology, WearableQA provides a stress test for LLMs that goes beyond simple pattern matching. The benchmark is designed to evaluate whether a model can actually "reason" about health, or if it is merely hallucinating plausible-sounding medical advice based on training data.

Technical Details

At the core of WearableQA is a dual-grounding framework. Constructing a dataset of this scale requires more than just raw data; it requires ground truth. The researchers combined literature-grounded physiological findings—established medical consensus—with statistically validated population-grounded physiological patterns derived from the data itself. This dual approach ensures that the questions are not only scientifically accurate but also contextually relevant to the specific data provided.

To categorize the reasoning capabilities required, the benchmark organizes its 16 question types along two primary axes:

1. Data vs. Health Reasoning: This axis distinguishes between computational tasks (e.g., "What was the average heart rate variability over the last 30 days?") and interpretative tasks (e.g., "Does this trend in resting heart rate suggest physiological stress?").
2. Single- vs. Cross-Signal Reasoning: This axis separates questions that focus on a single data stream (e.g., analyzing sleep duration) from those that require the integration of multiple signals (e.g., correlating sleep quality, physical activity, and blood glucose levels).

The dataset includes 10-option multiple-choice questions, which significantly reduces the probability of guessing correctly compared to standard 4-option benchmarks. By maintaining the authentic distributions of wearable data, the benchmark forces models to handle the noise and "missingness" that are ubiquitous in real-world deployments.

Benchmark Analysis

The initial evaluation of 14 proprietary and open-source models reveals a significant performance gap. While a 10% chance baseline exists, the top-performing models are far from achieving human-level reliability. The performance range across the tested models is stark, highlighting that even state-of-the-art architectures struggle with the nuances of longitudinal health data.

Most models failed to cross the 60% accuracy threshold. This suggests that while LLMs are proficient at retrieving medical facts, they lack the temporal reasoning and multi-modal integration skills necessary to act as reliable health assistants for longitudinal data analysis.

Developer Implications

For developers building AI-driven health applications, WearableQA serves as a diagnostic tool for model selection and fine-tuning. If your application relies on interpreting user trends over weeks or months, current off-the-shelf models may be insufficient. The benchmark demonstrates that "reasoning" in this context is not a monolithic capability; it is a composite of data computation and physiological interpretation.

Developers should consider the following when approaching this data:

• Temporal Context Window: Many LLMs struggle with long-context windows. If your data spans 500 days, ensure your architecture can effectively compress or attend to that history without losing critical signal.
• Noise Robustness: Real wearable data is noisy. Fine-tuning models on clean, synthetic data will likely result in poor performance in production. Incorporating noisy, real-world datasets into training pipelines is essential.
• Multi-Modal Integration: The cross-signal reasoning axis of the benchmark highlights that the most difficult questions involve correlating disparate data types. Models that treat signals in isolation will consistently fail these tasks.

Furthermore, the reliance on 10-option questions suggests that developers need to move away from binary "yes/no" or simple classification outputs. The complexity of health data requires models to select from a nuanced set of possibilities, reflecting the ambiguity often found in clinical settings.

Bottom Line

WearableQA exposes a critical vulnerability in current AI health applications: the inability to reason over longitudinal, noisy, real-world data. With most models failing to reach 60% accuracy, the industry is clearly in the early stages of developing AI capable of true physiological interpretation. For engineers, this benchmark provides a standardized way to measure progress, but it also serves as a reminder that we are still far from deploying autonomous, reliable health-reasoning agents.

Pneumetron