{"slug":"empirical-audit","label":"Empirical Audit","item_count":11,"day_count":7,"source_count":5,"first_seen":"2026-10-30T11:39:08+00:00","last_updated":"2026-11-16T16:19:01+00:00","generated_at":"2026-11-18T00:14:17.482304+00:00","sources":["arxiv_agent_systems_research","arxiv_cn_open_weight_reports","arxiv_cs_ai","arxiv_cs_cl","arxiv_llm_reliability"],"days":[{"date":"2026-10-30","items":[{"title":"Early Stopping in Large Reasoning Models: A Survey and Empirical Audit of Efficiency and Reliability","url":"http://arxiv.org/abs/2611.00166v1","source":"arxiv_llm_reliability","type":"paper","summary_1line":"Large reasoning models can improve accuracy by generating longer reasoning traces, but some of this computation may be unnecessary. Early-stopping methods aim to stop once enough reasoning is available to answer. Eval...","why_it_matters":"Matches feed focus: eval.","sid":"fdfc122cd97a630f","published":"2026-10-30T11:39:08+00:00","editor_note":"Survey plus audit of early-stopping methods for reasoning models, weighing saved compute against reliability."},{"title":"When Does Test-Time Search Help World Action Models? An Empirical Audit of Candidate Selection and Simulator Integrity","url":"http://arxiv.org/abs/2610.41970v1","source":"arxiv_cs_ai","type":"paper","summary_1line":"World-action models increasingly apply test-time scaling, spending extra inference computation on candidate selection and adaptive allocation to improve control. Such gains are credible only when candidates carry expl...","why_it_matters":"Matches feed focus: evaluation.","sid":"fd81b2702033c7c6","published":"2026-10-30T16:21:07+00:00","editor_note":"Asks when test-time search really helps world-action models, focusing on candidate selection and simulator integrity."},{"title":"Compression Technique Interactions in Small Language Models: A Factorial Ablation Study","url":"http://arxiv.org/abs/2611.00464v1","source":"arxiv_cs_ai","type":"paper","summary_1line":"Deploying small language models on resource-constrained hardware requires compression, yet practitioners lack systematic guidance on how quantization, pruning, and post-compression recovery interact when combined. We...","why_it_matters":"Matches feed focus: evaluation.","sid":"b80471ec3bc792a2","published":"2026-10-30T23:36:03+00:00","editor_note":"Factorial ablation of how quantization, pruning and recovery interact when stacked on small language models."}]},{"date":"2026-11-03","items":[{"title":"Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization","url":"http://arxiv.org/abs/2611.05373v1","source":"arxiv_cs_cl","type":"paper","summary_1line":"Direct Nash Optimization (DNO) aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. Its practical variant, DNO-Prct, iterates on-policy sampli...","why_it_matters":"Matches feed focus: evaluation.","sid":"5ab1b6ecf4ca3e7a","published":"2026-11-03T18:59:15+00:00","editor_note":"Empirical study of Direct Nash Optimization's practical iterative variant for general-preference alignment."}]},{"date":"2026-11-09","items":[{"title":"Dependable Long-Horizon LLM Agents: A Trajectory-Centered Empirical Study of Where Errors Enter and Which Controls Reach Them","url":"http://arxiv.org/abs/2611.13254v1","source":"arxiv_agent_systems_research","type":"paper","summary_1line":"When an LLM agent fails part-way through a long task, the usual analysis records a symptom: a failure-mode label, or the agent and step judged responsible. It does not say where the error entered, which state carried...","why_it_matters":"Matches feed focus: agent, harness.","sid":"94045c979b35ebe7","published":"2026-11-09T21:25:08+00:00","editor_note":"Traces where errors enter long-horizon agent trajectories and which controls can reach them, beyond failure-mode labels."}]},{"date":"2026-11-10","items":[{"title":"The AI Era's Lines of Code? A Case Study on What an Engineer's Token Spend Measures","url":"http://arxiv.org/abs/2611.13396v1","source":"arxiv_agent_systems_research","type":"paper","summary_1line":"Organizations use engineers' token spend to monitor AI adoption, set budgets, and measure productivity, but the metric's relationship with engineering output is unclear. We link five weeks of token usage in coding age...","why_it_matters":"Matches feed focus: agent, evaluation.","sid":"77f0b452ecfd74d0","published":"2026-11-10T00:17:26+00:00","editor_note":"Links five weeks of coding-agent token usage to engineering output to test token spend as a productivity metric."},{"title":"Can KV Cache Be Reused Across Model Updates? A DeepSeek-V4-Flash Case Study","url":"http://arxiv.org/abs/2611.14051v1","source":"arxiv_cn_open_weight_reports","type":"paper","summary_1line":"Reusing key--value (KV) caches across language model updates could avoid repeated computation of shared context, but changes in model weights raise a fundamental question about what these caches preserve. We investiga...","why_it_matters":"Matches feed focus: eval.","sid":"60cf435fff992a90","published":"2026-11-10T13:17:04+00:00","editor_note":"Tests whether KV caches stay reusable after a model update, using DeepSeek-V4-Flash."}]},{"date":"2026-11-11","items":[{"title":"Can We Make an LLM Dumber, or Is It Just Playing Dumb? An Empirical Study of Believable Performance Throttling","url":"http://arxiv.org/abs/2611.15568v1","source":"arxiv_agent_systems_research","type":"paper","summary_1line":"A benchmark score is commonly read as a proxy for model capability. We study the inverse problem: can a model be driven to a target score, and do its failures resemble those of a naturally weaker model? We operationalize believable performance reduction along three axes: target c","sid":"9b791542dc8bb7cc","published":"2026-11-11T14:53:50+00:00","editor_note":"Studies whether a model can be driven to a target benchmark score while its failures still look like a weaker model's."}]},{"date":"2026-11-13","items":[{"title":"Safety Side Effects of Preventative Persona Steering: An Empirical Audit","url":"http://arxiv.org/abs/2611.17620v1","source":"arxiv_llm_reliability","type":"paper","summary_1line":"High-level behaviors such as sycophancy and hallucination correspond to directions in a model's activation space (Chen et al., 2025). These directions, known as persona vectors, can suppress undesirable traits either at inference time, subtracting the vector during generation, or","why_it_matters":"Matches feed focus: evaluation.","sid":"44583beb4fa9105b","published":"2026-11-13T00:45:18+00:00","editor_note":"Audits safety side effects of preventative persona-vector steering against traits like sycophancy and hallucination."},{"title":"Can Mechanistic Interpretability Help Explain LLM Agent Failures? A Case Study on Web Element Hallucination","url":"http://arxiv.org/abs/2611.18514v1","source":"arxiv_llm_reliability","type":"paper","summary_1line":"Mechanistic Interpretability (MI) has made significant progress in explaining the internal computations of Large Language Models (LLMs); however, these advances have yet to demonstrate comparable success in explaining the decision-making processes of LLM-based agents. This paper","why_it_matters":"Matches feed focus: agentic.","sid":"75db6f155d9769a5","published":"2026-11-13T18:30:58+00:00","editor_note":"Case study applying mechanistic interpretability to web agents hallucinating page elements."}]},{"date":"2026-11-16","items":[{"title":"TRIAGE-CPG: A Cross-Model Study of Budgeted Static Malware Triage with Large Language Models","url":"http://arxiv.org/abs/2611.21331v1","source":"arxiv_cs_ai","type":"paper","summary_1line":"When triaging unknown binaries, security analysts must decide which samples to clear as benign, flag as malicious, or escalate for deeper analysis. Large Language Models (LLMs) could support these decisions by interpr...","why_it_matters":"Matches feed focus: agent, eval.","sid":"8f0034965a63d29c","published":"2026-11-16T16:19:01+00:00","editor_note":"Extends the audit pattern to security: compares LLMs on clear/flag/escalate decisions for unknown binaries under a budget."}]}],"editorial":{"tldr":"This is a research trend, not one developing event: eleven unrelated arXiv papers from Oct 30 to Nov 16 that each re-test a technique already in use. It began with audits of inference cost (early stopping, test-time search, small-model compression) and spread to alignment, token-spend metrics, KV-cache reuse, benchmark integrity and agent debugging.","stale":false,"whats_new":"On Nov 16 the audits reached security: TRIAGE-CPG compares several LLMs on budgeted static malware triage, testing which models can clear, flag or escalate unknown binaries under a fixed analysis budget.","why_it_matters":"Each paper re-checks an assumption platform teams already ship on (early stopping, compression stacks, token budgets, cache reuse, steering vectors, benchmark scores, LLM triage), so the findings feed existing eval and cost decisions rather than adding features.","take_for_builders":"Before tuning early stopping, compression, token budgets, cross-version KV-cache reuse or steering vectors in production, read the matching audit and rerun its check on your own workload rather than trusting vendor defaults.","beats":[{"kicker":"COMPUTE","tone":"launch","headline":"Audits of inference cost: early stopping, test-time search and compression","summary":"Three papers ask whether cheaper or extra inference actually buys what it promises.","sids":["fdfc122cd97a630f","fd81b2702033c7c6","b80471ec3bc792a2"]},{"kicker":"ALIGNMENT","tone":"neutral","headline":"Direct Nash Optimization tested as a practical alignment recipe","sids":["5ab1b6ecf4ca3e7a"]},{"kicker":"COST & INFRA","tone":"rising","headline":"Token spend as a productivity metric, and KV caches across model updates","summary":"Two operational assumptions examined: what token budgets measure and whether cached context survives a weight update.","sids":["77f0b452ecfd74d0","60cf435fff992a90"]},{"kicker":"EVAL INTEGRITY","tone":"turn","headline":"Benchmark scores can be throttled, and persona steering has safety side effects","summary":"Two papers question signals teams use to judge model capability and safety.","sids":["9b791542dc8bb7cc","44583beb4fa9105b"]},{"kicker":"AGENT FAILURES","tone":"rising","headline":"Where errors enter long agent trajectories, and whether interpretability can explain them","sids":["94045c979b35ebe7","75db6f155d9769a5"]},{"kicker":"NOW","tone":"now","headline":"Cross-model audit of LLMs for budgeted malware triage","sids":["8f0034965a63d29c"]}],"open_questions":["Do the early-stopping and test-time search audits publish thresholds a serving stack can adopt directly?","Does the compression ablation name a quantization-plus-pruning order that is safe to combine on small models?","Does the DeepSeek-V4-Flash KV-cache result hold for other model families or larger weight updates?","Does any provider or eval harness add checks for deliberate score throttling (sandbagging)?","Does the trajectory-centered error study release its traces or controls so other agent teams can reproduce it?","Does TRIAGE-CPG release its benchmark so security teams can rerun the model comparison on their own samples?"],"generated_at":"2026-11-18T00:13:28Z"}}