LLM Digest
Subscribe

AI Storyline

2 items · 2 sources · 2 days

View as JSON

Operational story trace

OpenAI fixes 18-year-old libunwind core-dump bug

Current stateResolved · fix upstreamedstatus changed Jul 9

Latest change

InfoQ's write-up details the fix: an 18-year-old GNU libunwind race condition with a roughly 100-picosecond vulnerable window, made real by OpenAI's frequent per-query timer signals, has been patched upstream by reordering the instruction- and stack-pointer updates.

Earlier contextThe story so far

OpenAI engineers spent months chasing mysterious crashes in Rockset, the C++ data infrastructure behind ChatGPT search, where functions returned to invalid addresses and stack pointers shifted unexpectedly. Instead of debugging crashes one at a time, they built a pipeline to mine a year of production core dumps for population-level patterns — "epidemiological debugging."

editor-curated · source-linked

Arc

Jun 30Jul 9 · now
INVESTIGATION · Jun 30
OpenAI mines a year of ChatGPT core dumps to catch untraceable Rockset crashes
1 source · scout · show source ▾
THE FIX · Jul 9
Root cause: a silent Azure hardware fault plus an 18-year-old libunwind race condition
The software bug's vulnerable window was about 100 picoseconds — only exposed by OpenAI's frequent per-query timer signals — and was closed by reordering the pointer updates.
1 source · scout · show source ▾

What to watch — open questions

  • Has the libunwind race-condition fix landed in a released version, or only been submitted upstream?
  • Did the same Azure hardware fault affect other tenants on that host, or was it isolated to OpenAI's workload?
How this thread was built
scout surfaced 2editor wrote the arc · 2 beatswatcher 1 status change

Storylines are threaded mechanically from the feed: stories that share a distinctive anchor across multiple days and sources. Each item links to its original source. The evidence trace, current state, and open questions are written by the editor routine and refreshed whenever a new beat lands.