B
Bruno Walter
Guest
We have a C# app compiled in VS2010 against .NET 4.0 that loads many assemblies, some mixed mode, in a new app domain. Occasionally our clients notice "zombie" processes of our app. Each zombie processes corresponds to an ASYNC_NETWORK_IO waits in the clients SQL server. A process dump shows a hang in one of our mixed mode assemblies (the specific assembly varies).
Based on the calls stack and logs it appears the process is being shut down (presumably because the client canceled the login). Its worth noting that I havent managed to reproduce the problem myself and the client only sees the issue in their production environment, not their development and test environments.
Call stack is as follows:
ntdll.dll!_NtDelayExecution@8() + 0x15 bytes
ntdll.dll!_NtDelayExecution@8() + 0x15 bytes
KERNELBASE.dll!_Sleep@4() + 0xf bytes
> OurLib.Context.dll!_CRT_INIT(void * hDllHandle=0x05dc0000, unsigned long dwReason=0, void * lpreserved=0x00000001) Line 357 C
OurLib.Context.dll!__DllMainCRTStartup(void * hDllHandle=0x05dc0000, unsigned long dwReason=0, void * lpreserved=0x00000000) Line 526 + 0x8 bytes C
OurLib.Context.dll!_DllMainCRTStartup(void * hDllHandle=0x05dc0000, unsigned long dwReason=0, void * lpreserved=0x00000001) Line 476 + 0xe bytes C
mscoreei.dll!__CorDllMain@12() + 0xde bytes
mscoree.dll!_ShellShim__CorDllMain@12() + 0xad bytes
ntdll.dll!_LdrpCallInitRoutine@16() + 0x14 bytes
ntdll.dll!_LdrShutdownProcess@0() + 0x141 bytes
ntdll.dll!_RtlExitUserProcess@4() + 0x74 bytes
kernel32.dll!76d179ed()
mscoreei.dll!RuntimeDesc::ShutdownAllActiveRuntimes() + 0xc8 bytes
mscoreei.dll!CLRRuntimeHostInternalImpl::ShutdownAllRuntimesThenExit() + 0x15 bytes
clr.dll!EEPolicy::ExitProcessViaShim() + 0x66 bytes
clr.dll!SafeExitProcess() + 0x99 bytes
clr.dll!DisableRuntime() - 0x160f1b bytes
clr.dll!EEPolicy::HandleExitProcess() + 0x57 bytes
clr.dll!__CorExeMainInternal@0() + 0x11c bytes
clr.dll!__CorExeMain@0() + 0x1c bytes
mscoreei.dll!__CorExeMain@0() + 0x38 bytes
mscoree.dll!_ShellShim__CorExeMain@0() + 0x227 bytes
mscoree.dll!__CorExeMain_Exported@0() + 0x8 bytes
kernel32.dll!@BaseThreadInitThunk@12() + 0x12 bytes
ntdll.dll!___RtlUserThreadStart@8() + 0x27 bytes
ntdll.dll!__RtlUserThreadStart@8() + 0x1b bytes
The _CRT_INIT method is in crtdll.c, namely it appears to be the sleep in the code below:
else if ( dwReason == DLL_PROCESS_DETACH )
{
/*
* Any basic clean-up code that goes here must be
* duplicated below in _DllMainCRTStartup for the
* case where the users DllMain() routine fails on a
* Process Attach notification. This does not include
* calling user C++ destructors, etc.
*/
/*
* do _onexit/atexit() terminators
* (if there are any)
*
* These terminators MUST be executed in
* reverse order (LIFO)!
*
* NOTE:
* This code assumes that __onexitbegin
* points to the first valid onexit()
* entry and that __onexitend points
* past the last valid entry. If
* __onexitbegin == __onexitend, the
* table is empty and there are no
* routines to call.
*/
void *lock_free=0;
void *fiberid=((PNT_TIB)NtCurrentTeb())->StackBase;
int nested=FALSE;
while((lock_free=InterlockedCompareExchangePointer((volatile PVOID *)&__native_startup_lock, fiberid, 0))!=0)
{
if(lock_free==fiberid)
{
nested=TRUE;
break;
}
/* some other thread is running native startup/shutdown during a cctor/domain unload.
Should only happen if this DLL was built using the Everett-compat loader lock fix in vcclrit.h
*/
/* wait for the other thread to complete init before we return */
Sleep(1000);
}
The comment suggests some other thread has a lock, however based on the dump, the only thread remaining is the main thread. Since the thread which has the lock no longer exists, the code ends up stuck in this loop. Im really hoping someone can provide some insight as to what the underlying issue could be. This looks like a form of loader lock to me, but it doesnt appear involve DllMain or other code that I am directly in control of so Im not sure what I can do to attempt to resolve it.
Continue reading...
Based on the calls stack and logs it appears the process is being shut down (presumably because the client canceled the login). Its worth noting that I havent managed to reproduce the problem myself and the client only sees the issue in their production environment, not their development and test environments.
Call stack is as follows:
ntdll.dll!_NtDelayExecution@8() + 0x15 bytes
ntdll.dll!_NtDelayExecution@8() + 0x15 bytes
KERNELBASE.dll!_Sleep@4() + 0xf bytes
> OurLib.Context.dll!_CRT_INIT(void * hDllHandle=0x05dc0000, unsigned long dwReason=0, void * lpreserved=0x00000001) Line 357 C
OurLib.Context.dll!__DllMainCRTStartup(void * hDllHandle=0x05dc0000, unsigned long dwReason=0, void * lpreserved=0x00000000) Line 526 + 0x8 bytes C
OurLib.Context.dll!_DllMainCRTStartup(void * hDllHandle=0x05dc0000, unsigned long dwReason=0, void * lpreserved=0x00000001) Line 476 + 0xe bytes C
mscoreei.dll!__CorDllMain@12() + 0xde bytes
mscoree.dll!_ShellShim__CorDllMain@12() + 0xad bytes
ntdll.dll!_LdrpCallInitRoutine@16() + 0x14 bytes
ntdll.dll!_LdrShutdownProcess@0() + 0x141 bytes
ntdll.dll!_RtlExitUserProcess@4() + 0x74 bytes
kernel32.dll!76d179ed()
mscoreei.dll!RuntimeDesc::ShutdownAllActiveRuntimes() + 0xc8 bytes
mscoreei.dll!CLRRuntimeHostInternalImpl::ShutdownAllRuntimesThenExit() + 0x15 bytes
clr.dll!EEPolicy::ExitProcessViaShim() + 0x66 bytes
clr.dll!SafeExitProcess() + 0x99 bytes
clr.dll!DisableRuntime() - 0x160f1b bytes
clr.dll!EEPolicy::HandleExitProcess() + 0x57 bytes
clr.dll!__CorExeMainInternal@0() + 0x11c bytes
clr.dll!__CorExeMain@0() + 0x1c bytes
mscoreei.dll!__CorExeMain@0() + 0x38 bytes
mscoree.dll!_ShellShim__CorExeMain@0() + 0x227 bytes
mscoree.dll!__CorExeMain_Exported@0() + 0x8 bytes
kernel32.dll!@BaseThreadInitThunk@12() + 0x12 bytes
ntdll.dll!___RtlUserThreadStart@8() + 0x27 bytes
ntdll.dll!__RtlUserThreadStart@8() + 0x1b bytes
The _CRT_INIT method is in crtdll.c, namely it appears to be the sleep in the code below:
else if ( dwReason == DLL_PROCESS_DETACH )
{
/*
* Any basic clean-up code that goes here must be
* duplicated below in _DllMainCRTStartup for the
* case where the users DllMain() routine fails on a
* Process Attach notification. This does not include
* calling user C++ destructors, etc.
*/
/*
* do _onexit/atexit() terminators
* (if there are any)
*
* These terminators MUST be executed in
* reverse order (LIFO)!
*
* NOTE:
* This code assumes that __onexitbegin
* points to the first valid onexit()
* entry and that __onexitend points
* past the last valid entry. If
* __onexitbegin == __onexitend, the
* table is empty and there are no
* routines to call.
*/
void *lock_free=0;
void *fiberid=((PNT_TIB)NtCurrentTeb())->StackBase;
int nested=FALSE;
while((lock_free=InterlockedCompareExchangePointer((volatile PVOID *)&__native_startup_lock, fiberid, 0))!=0)
{
if(lock_free==fiberid)
{
nested=TRUE;
break;
}
/* some other thread is running native startup/shutdown during a cctor/domain unload.
Should only happen if this DLL was built using the Everett-compat loader lock fix in vcclrit.h
*/
/* wait for the other thread to complete init before we return */
Sleep(1000);
}
The comment suggests some other thread has a lock, however based on the dump, the only thread remaining is the main thread. Since the thread which has the lock no longer exists, the code ends up stuck in this loop. Im really hoping someone can provide some insight as to what the underlying issue could be. This looks like a form of loader lock to me, but it doesnt appear involve DllMain or other code that I am directly in control of so Im not sure what I can do to attempt to resolve it.
Continue reading...
