OpenAI details debugging of two infrastructure crashes

OpenAI says it identified and fixed two unrelated causes behind unusual crashes in its Rocksetbased data infrastructure: a faulty Azure host and an 18yearold race condition in GNU libunwind. The company said the investigation began after repeated C++ crashes in Rockset’s queryprocessing layer appeared to corrupt stack state or return to invalid addresses. Initial manual inspection of a few core dumps did not reveal a clear pattern, so the team built a pipeline to analyze production crash data at scale. By classifying the full set of core dumps from the previous year, engineers found that one group of crashes was tied to a single physical host and disappeared after that machine was removed from service. A second group was linked to exception unwinding in GNU libunwind, where a narrow race condition could let a signal handler overwrite stackallocated unwind data and corrupt the instruction pointer during control transfer. OpenAI said it mitigated the issue by switching to libgcc’s unwinder, upstreaming a reproducer and fix to libunwind, and improving logging and operational safeguards to make similar incidents easier to detect in the future.