HeadlinesBriefing favicon HeadlinesBriefing.com

NVIDIA nvidia-smi Hang Bug After 66 Days

Hacker News: Front Page •
×

A bug report on NVIDIA's open-gpu-kernel-modules GitHub repo details nvidia-smi hanging indefinitely after approximately 66 days of uptime. The issue occurs on driver 570.133.20 using the OpenRM stack on a B200 GPU with Linux kernel 6.6.0. The user confirmed the proprietary driver of the same version does not exhibit this behavior, isolating the problem to the open-source kernel module.

The reporter provided system logs showing repeated NVRM errors related to knvlink link mask updates failing. This points to a potential race condition or resource exhaustion in the NVLink management code after extended operation. Such long-running stability issues are critical for data centers and AI training clusters where GPUs run for weeks or months without interruption.

NVIDIA has triaged the issue and created an internal NVBug for investigation. The community is seeking more detailed logs and stack traces to diagnose the hang. This bug highlights the ongoing maturation of NVIDIA's open-source driver effort and the importance of long-term stability testing for enterprise-grade GPU workloads.