The case of the UI thread that hung in a kernel call

Summary

A customer asked for help with a longstanding but low-frequency hang that they have never been able to figure out. From what they could tell, their UI thread was calling into the kernel, and the call simply hung for no apparent reason. Unfortunately, the kernel dump couldn’t show a stack from user mode because the stack had been paged out. (Which makes sense, because a hung thread isn’t using its stack, so once the system is under some memory pressure, that stack gets paged out.) 0: kd> !thread 0xffffd18b976ec080 7 THREAD ffffd18b976ec080 Cid 79a0.7f18 Teb: 0000003d7ca28000 Win32Thread: ffffd18b89a8f170 WAIT: (Suspended) KernelMode Non-Alertable SuspendCount 1 ffffd18b976ec360 NotificationEvent Not impersonating DeviceMap ffffad897944d640 Owning Process ffffd18bcf9ec080 Image: contoso.exe Attached Process N/A Image: N/A Wait Start TickCount 14112735 Ticks: 1235580 (0:05:21:45.937) Context Switch Count 1442664 IdealProcessor: 2 UserTime 00:02:46.015 KernelTime 00:01:11.515 nt!KiSwapContext+0x76 nt!KiSwapThread+0x928 nt!KiCommitThreadWait+0x370 nt!KeWaitForSingleObject+0x7a4 nt!KiSchedulerApc+0xec nt!KiDeliverApc+0x5f9 nt!KiCheckForKernelApcDelivery+0x34 nt!MiUnlockAndDereferenceVad+0x8d nt!MmProtectVirtualMemory+0x312 nt!NtProtectVirtualMemory+0x1d9 nt!KiSystemServiceCopyEnd+0x25 (TrapFrame @ ffff8707`a9bef3a0) ntdll!ZwProtectVirtualMemory+0x14 [end of stack trace] Although we couldn’t see what the code was doing in user mode, there was something unusual in the information that was present. Observe that the offending thread is Suspended. And it appears to have been suspended for over five hours. THREAD ffffd18b976ec080 Cid 79a0.7f18 Teb: 0000003d7ca28000 Win32Thread: ffffd18b89a8f170 WAIT: (Suspended) KernelMode Non-Alertable SuspendCount 1 ffffd18b976ec360 NotificationEvent Not impersonating DeviceMap ffffad897944d640 Owning Process ffffd18bcf9ec080 Image: contoso.exe Attached Process N/A Image: N/A Wait Start TickCount 14112735 Ticks: 1235580 (0:05:21:45.937) Natur...