HeadlinesBriefing favicon HeadlinesBriefing.com

Debugging a Mysterious DLL Crash: Stack Overflow Investigation

Hacker News •
×

Microsoft's shell32.dll team investigated crash reports from a third-party application, discovering a stack overflow caused by recursive exception handling. The crash dumps revealed an endless loop between RtlLookupFunctionEntry and KiUserExceptionDispatch, consuming the process stack until hitting the overflow limit. This type of debugging requires understanding Windows exception dispatch mechanisms and how kernel-mode exceptions get reflected back to user mode for processing.

The investigation traced the original exception to combase!CoTaskMemFree, which triggered an access violation when attempting to execute code at a non-executable address. Surprisingly, the memory analysis showed combase.dll was marked as MEM_FREE despite never being formally unloaded, suggesting a corrupted module state. The crash occurred during process shutdown, involving the CLR runtime and multiple DLL detach operations.

Root cause analysis revealed the DLL's memory had been freed or corrupted, making its code pages inaccessible. When CoTaskMemFree tried to execute, the CPU raised an access violation because the page lacked execute permissions. This created the recursive exception scenario that exhausted the stack.

The case demonstrates how Windows module lifecycle management can break down during shutdown sequences, creating scenarios where valid code pointers reference invalid memory. Debugging such issues requires examining both stack traces and memory state to understand the full picture.