Neural scaling law breakdowns and irreducible loss

Every dataset carries irreducible entropy — inherent unpredictability in the next token (ambiguous context, multiple valid continuations, label noise). A scaling law's loss curve asymptotes toward this entropy floor, $L(N) \to L_\infty$, as reducible loss shrinks to zero. No amount of extra parameters, data, or compute pushes loss below $L_\infty$: you're not fighting a solvable prediction problem anymore, you're fighting the genuine randomness baked into the data-generating process itself.