LLM Digest

Story

arxiv_cs_cl · Nov 2, 2026 · paper

Source brief

Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks

arxiv.orgNov 2, 2026
original source linked

In brief

Every multimodal-LLM multiple-choice benchmark ends with a scoring rule that converts a model's response, often free-form text rather than a clean option letter, into a right/wrong verdict, buried in its code. Unlike the dataset, this rule is chosen without justification and left

Read the original at arxiv.org →Open in live feed Read that day’s brief

Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks

Earlier in this thread 4 items

Same model, different agent: 17 points apart on 21 MCP servers

Same Story, Different Lens: A Multi-Agent System for Cross-Outlet News Framing Analysis

Same Luck, Different Choice: Twin-World Credit Assignment for Agentic Reinforcement Learning

Same molecule, Different Knowledge: Cross-Identifier Binding Failures in LLMs For chemistry