Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases
Opening argument: static correctness checks don't capture real diagnostic reasoning.
AI Storyline
3 items · 3 sources · 3 days
Operational story trace
Evaluating Reasoning
Follow in this browser to see new updates on your Live feed.
Latest change
Databricks ran the first live "Grounded Reasoning Cup," testing AI agents on grounded tasks in real time rather than against a fixed offline benchmark.
Earlier contextThe story so far
A clinical-diagnosis benchmark argued reasoning evals need multi-turn, progressively-disclosed cases instead of single-shot correctness checks. Two weeks later, VAKRA extended that critique to agents, scoring API calls and retrieval together instead of in isolation.
editor-curated · source-linked
Arc
Jul 28Aug 18 · now
THE CRITIQUE · Jul 28Clinical benchmark says correctness checks miss real diagnostic reasoning1 source · show source ▾
WIDENING SCOPE · Aug 12VAKRA benchmarks agents across APIs and retrieval together, not in isolation1 source · show source ▾
VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies
Extends the critique to agents, introducing a benchmark that scores API and retrieval reasoning jointly.
NOW · Aug 18Databricks runs the first live "Grounded Reasoning Cup" for AI agents1 source · show source ▾
Evaluating AI Agents Live at the Grounded Reasoning Cup
First live, grounded evaluation event puts the critique into practice outside offline benchmarks.
Day 1 Tuesday, Jul 28, 2026
Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases
Opening argument: static correctness checks don't capture real diagnostic reasoning.
Day 2 Wednesday, Aug 12, 2026
VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies
Extends the critique to agents, introducing a benchmark that scores API and retrieval reasoning jointly.
Day 3 Tuesday, Aug 18, 2026
Evaluating AI Agents Live at the Grounded Reasoning Cup
First live, grounded evaluation event puts the critique into practice outside offline benchmarks.
What to watch — open questions
- Will Databricks publish the Grounded Reasoning Cup's task set and agent rankings for reproducibility?
- Do VAKRA's cross-API findings hold for agents built on orchestration frameworks outside its own harness?
- Does progressive-disclosure, multi-turn evaluation spread beyond clinical use cases into general agent benchmarks?
How this thread was built
editor wrote the arc · 3 beats
Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.