LLM Digest

Story

arxiv_cs_lg · Nov 9, 2026 · paper

Source brief

What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts

arxiv.orgNov 9, 2026
original source linked

In brief

Low-precision pretraining is usually judged by validation loss. We ask whether matching loss also means learning rare facts as efficiently. We train transformers from scratch with emulated 4-bit matrix multiplications...

Feed lens

eval

Read the original at arxiv.org →Open in live feed Read that day’s brief

What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts

Earlier in this thread 4 items

What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models

From Twisted Facts to Rogue Agents: Can LLMs Reliably Detect Multi-Dimensional Hallucinations?

A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out

Scale-Aware Pretraining of Time Series Foundation Models via Multi-Patch Token Alignment and Hybrid Masking