The Most Common Questions About Agent Evals, Answered
LangChain's FAQ covers offline vs online evals, LLM judges, multi-turn evals and Harbor benchmarks, with tips for keeping eval cost down.
AI Daily Recap
25 articles · 4 categories
The finishable daily brief
What happened in AI — Oct 15, 2026
Thursday, Oct 15, 2026
25 articles · 4 categories
read top to bottom · then stop
In 30 seconds
- LangChain answers common agent-eval questions: offline vs online, LLM judges, multi-turn, cost control.
- AWS Trust in Depth: scope, know, bound, observe, prove for agent autonomy.
- LangChain ships LangSmith AI Search, Trajectories and Managed Deep Agents v0.9.
- Modal hits $1B ARR on a $16B Series D as sandboxes become core infra.
- Meta Muse Spark Open lands as a top U.S. open-weights model; vLLM adds Nemotron 3.5 Super VL day 0.
- DeepSeek down again, with refunds offered.
Agent builders got concrete guidance on evals and bounded autonomy from LangChain, AWS and Anthropic, while LangChain and Modal showed the tooling and sandbox layer maturing.
Meta's open-weights release and vLLM's day-0 Nemotron support moved the serving story, and DeepSeek's second major outage was a reminder to plan for provider failover.
Agent Evals and Trust Boundaries 3 items
Three vendors published practical guidance on measuring agents and bounding their autonomy rather than trusting them blindly.
Trust in Depth: Giving agents more autonomy without losing control
AWS's Trust in Depth frames agent autonomy as five layers: scope, know, bound, observe, prove.
Anthropic update on independent evaluators
Anthropic updates how it embeds independent evaluators into its model assessment process.
Agent Platforms and Developer Tooling 4 items
Agent infrastructure vendors shipped tracing, search and managed runtimes, while sandboxes drew major capital.
October 2026 LangChain Newsletter: Everything we shipped at Interrupt NYC & London
LangChain ships LangSmith AI Search, Engine v2, Trajectories, Custom Apps, fine-tuning and Managed Deep Agents v0.9.
[AINews] Modal’s $1B ARR, $16B Series D makes it NYC’s newest AI Infra Decacorn, 3x usage and 3x valuation in 5 months, 15x in 1 yr
Modal reaches $1B ARR and a $16B Series D, with usage up 3x in five months, a signal that agent sandboxes are now core infrastructure.
Codex CLI Defaults to GPT-6.1 Sol, 77% Cheaper [2026]
Codex CLI reportedly defaults to GPT-6.1 Sol at a 77% lower price; the report is from a secondary site, so confirm against OpenAI's docs.
Show HN: Reproducible failure drills for AI coding agents
A Show HN project offers reproducible failure drills for testing how coding agents break.
Open Weights and Inference Serving 3 items
Open-weight model news centered on Meta's new release and same-day serving support in vLLM.
Meta unveils Muse Spark Open, the new top U.S. open weights model — and it runs on two paired Mac desktops
Meta's Muse Spark Open is billed as the top U.S. open-weights model and runs on two paired Mac desktops.
Announcing Day 0 Support for NVIDIA Nemotron 3.5 Super VL on vLLM
vLLM adds day-0 support for NVIDIA Nemotron 3.5 Super VL, with OpenAI-compatible serving and multi-token prediction for multimodal agent workflows.
FLock.io Launches Qwen-powered, Open-Source Decision Model for Local Edge Deployment
FLock.io releases an open-source, Qwen-based decision model aimed at local edge deployment.
Provider Reliability and Adoption Signals 4 items
A DeepSeek outage and new adoption data show where API dependence and trust bite.
DeepSeek suffers another major outage as company offers user refunds
DeepSeek suffered a second major outage and is offering user refunds.
DeepSeek Went Down for 5 Hours. The Silence Afterward Is the Real Story.
A post-mortem-style analysis notes DeepSeek was down about 5 hours and faults the silence that followed.
Presentation: What 25 Trillion Tokens Reveal About How Developers Actually Adopt AI Agents
InfoQ talk draws on usage data across tens of trillions of tokens: agent adoption hinges on trust, not benchmark chasing.
Anthropic tells investors quarterly revenue up nearly 50% to $16.8bn ahead of IPO
Anthropic told investors quarterly revenue rose nearly 50% to $16.8bn ahead of a planned IPO.
You are caught up for this edition