My entire process freezes. - Help. -- dotnet clr

esafran
3/19/2007 3:26:20 AM

Hi,

I experience a process freeze on my MultiThreaded console application.
(none of the threads are working)
When the process freeze occurs, I take a hang dump and I always see 2
of my threads in the same place.
Each thread is in a constructor, (different .ctor in each thread),
there is no locking or synchronization mechanism between those 2
threads.

The problem is, I can run the process number of times and in some
cases these threads will pass the code section where they usually hang
the entire process and in that case, everything will work fine.
But, sometimes, they will get stuck in the .ctors and the entire
process will hang.

I've taken a !dumpstack of one of the hanged threads (thread 57), and
I'm attaching it here from the part where I last seen my code
involvement.

i really would like to know on what handle my thread is waiting
(WaitForSingleObject):

Thread 57
Current frame: ntdll!KiFastSystemCallRet
ChildEBP RetAddr Caller,Callee
0dd9d4f4 7c90e9c0 ntdll!ZwWaitForSingleObject+0xc
0dd9d4f8 7c91901b ntdll!RtlpWaitForCriticalSection+0x132, calling
ntdll!NtWaitForSingleObject
0dd9d518 7c911970 ntdll!RtlpFreeDebugInfo+0x6a, calling ntdll!
_SEH_epilog
0dd9d52c 7c9105c8 ntdll!RtlpFreeToHeapLookaside+0x22, calling ntdll!
RtlpInterlockedPushEntrySList
0dd9d538 7c910551 ntdll!RtlFreeHeap+0x1e9, calling ntdll!
RtlpFreeToHeapLookaside
0dd9d540 7c91056d ntdll!RtlFreeHeap+0x647, calling ntdll!_SEH_epilog
0dd9d56c 009f15bb fusion!operator delete+0x1b, calling ntdll!
RtlFreeHeap
0dd9d580 7c90104b ntdll!RtlEnterCriticalSection+0x46, calling ntdll!
RtlpWaitForCriticalSection
0dd9d588 7c917332 ntdll!LdrUnloadDll+0x38, calling ntdll!
RtlEnterCriticalSection
0dd9d5a8 009f1c26 fusion!CAssemblyName::`scalar deleting
destructor'+0x17, calling fusion!operator delete
0dd9d5cc 792017ea mscorwks!AssemblySpec::LowLevelLoadManifestFile
+0x3b5, calling msvcr71!free
0dd9d5d0 79217931 mscorwks!AssemblySpec::LowLevelLoadManifestFile
+0x3d1, calling mscorwks!__security_check_cookie
0dd9d600 7c91056d ntdll!RtlFreeHeap+0x647, calling ntdll!_SEH_epilog
0dd9d604 7c80995a kernel32!LocalFree+0x27, calling ntdll!RtlFreeHeap
0dd9d610 7c80996d kernel32!LocalFree+0x108, calling kernel32!
_SEH_epilog
0dd9d628 7c910e91 ntdll!RtlFreeHeap+0x4fc, calling ntdll!
RtlLeaveCriticalSection
0dd9d648 7c80996d kernel32!LocalFree+0x108, calling kernel32!
_SEH_epilog
0dd9d64c 7921a391 mscorwks!EEUnicodeHashTableHelper::DeleteEntry+0xc,
calling kernel32!LocalFree
0dd9d674 7c80abf7 kernel32!FreeLibrary+0x3f, calling ntdll!
LdrUnloadDll
0dd9d688 7925cbdf mscorwks!CorMap::ReleaseHandleResources+0x70,
calling kernel32!FreeLibrary
0dd9d6a0 7921a380 mscorwks!CorMapInfo::Release+0x63, calling mscorwks!
CorMap::ReleaseHandleResources
0dd9d6b8 791c592e mscorwks!CorMap::ReleaseHandle+0xe, calling mscorwks!
CorMapInfo::Release
0dd9d6c0 791c6328 mscorwks!PEFile::~PEFile+0x37, calling mscorwks!
CorMap::ReleaseHandle
0dd9d6cc 791c6335 mscorwks!PEFile::`scalar deleting destructor'+0xb,
calling mscorwks!PEFile::~PEFile
0dd9d6d8 79218410 mscorwks!BaseDomain::LoadAssembly+0x28b, calling
mscorwks!PEFile::`scalar deleting destructor'
0dd9d6f0 7c91056d ntdll!RtlFreeHeap+0x647, calling ntdll!_SEH_epilog
0dd9d6f4 7c34218a msvcr71!free+0xc3, calling ntdll!RtlFreeHeap
0dd9d700 7c34218f msvcr71!free+0xc8, calling msvcr71!_SEH_epilog
0dd9d738 7c34218f msvcr71!free+0xc8, calling msvcr71!_SEH_epilog
0dd9d73c 791e9ccb mscorwks!EEClass::_GetFullyQualifiedNameForClass
+0xc3, calling msvcr71!free
0dd9d74c 791e9ce3 mscorwks!EEClass::_GetFullyQualifiedNameForClass
+0xdb, calling mscorwks!__security_check_cookie
0dd9d760 793852c5 mscorwks!MDInternalRW::GetFieldDefProps+0x51,
calling mscorwks!CMDSemReadWrite::~CMDSemReadWrite
0dd9d78c 791e909a mscorwks!CEEInfo::getFieldAttribs+0xd0, calling
mscorwks!FieldDesc::IsSpecialStatic
0dd9d7a0 79431a6d mscorjit!Compiler::impAppendStmt+0xae, calling
mscorjit!Compiler::impCurStmtOffsSet
0dd9d7b8 79431a8f mscorjit!Compiler::impAppendTree+0x1b, calling
mscorjit!Compiler::impAppendStmt
0dd9d7c8 79431daf mscorjit!Compiler::impImportBlockCode+0x3b8f,
calling mscorjit!Compiler::impAppendTree
0dd9d7ec 791b2fa2 mscorwks!AutoCooperativeGC::AutoCooperativeGC+0x18,
calling mscorwks.pdb not exist
Use alternate method which may not work.
00a71e90
0dd9d7f8 791b5655 mscorwks!HashMap::LookupValue+0x22, calling mscorwks!
AutoCooperativeGC::AutoCooperativeGC
0dd9d7fc 791b56ce mscorwks!HashMap::LookupValue+0xbb, calling mscorwks!
AutoCooperativeGC::~AutoCooperativeGC
0dd9d84c 792176e6 mscorwks!AssemblySpec::LoadAssembly+0x4da, calling
mscorwks!BaseDomain::LoadAssembly
0dd9d880 791e9411 mscorwks!EEClassHashTable::ConstructKeyFromData+0x43
0dd9d8b4 791e3aee mscorwks!EEClassHashTable::CompareKeys+0x5c, calling
mscorwks!EEClassHashTable::ConstructKeyFromData
0dd9d8b8 791e3b4c mscorwks!EEClassHashTable::CompareKeys+0x8c, calling
msvcr71!strcmp
0dd9d92c 7c91056d ntdll!RtlFreeHeap+0x647, calling ntdll!_SEH_epilog
0dd9d930 7c34218a msvcr71!free+0xc3, calling ntdll!RtlFreeHeap
0dd9d93c 7c34218f msvcr71!free+0xc8, calling msvcr71!_SEH_epilog
0dd9d974 7c34218f msvcr71!free+0xc8, calling msvcr71!_SEH_epilog
0dd9d978 791e9b81 mscorwks!
MarshalInfo::OutputCustomerCheckedBuildMarshalInfo+0x41a, calling
msvcr71!free
0dd9d984 791e9bac mscorwks!
MarshalInfo::OutputCustomerCheckedBuildMarshalInfo+0x44d, calling
mscorwks!__security_check_cookie
0dd9d98c 51a92309 diasymreader!SymCachePdb::mapModIdToGlobalId+0x67,
calling diasymreader!__security_check_cookie
0dd9d9a8 7c910732 ntdll!RtlpAllocateFromHeapLookaside+0x42, calling
ntdll!_SEH_epilog
0dd9d9d4 7c910732 ntdll!RtlpAllocateFromHeapLookaside+0x42, calling
ntdll!_SEH_epilog
0dd9d9e0 7c910732 ntdll!RtlpAllocateFromHeapLookaside+0x42, calling
ntdll!_SEH_epilog
0dd9da10 7c911596 ntdll!RtlAllocateHeap+0x43d, calling ntdll!
RtlLeaveCriticalSection
0dd9da14 7c9106eb ntdll!RtlAllocateHeap+0xeac, calling ntdll!
_SEH_epilog
0dd9da24 7c910732 ntdll!RtlpAllocateFromHeapLookaside+0x42, calling
ntdll!_SEH_epilog
0dd9da50 7c910732 ntdll!RtlpAllocateFromHeapLookaside+0x42, calling
ntdll!_SEH_epilog
0dd9da54 7c9106ab ntdll!RtlAllocateHeap+0x1c2, calling ntdll!
RtlpAllocateFromHeapLookaside
0dd9da58 7c9106eb ntdll!RtlAllocateHeap+0xeac, calling ntdll!
_SEH_epilog
0dd9da70 794369ca mscorjit!Compiler::genCodeForTree_DONE+0xd, calling
mscorjit!Compiler::genUpdateLife
0dd9da80 7943e103 mscorjit!Compiler::genCodeForTree_REG_VAR1+0x24,
calling mscorjit!Compiler::genCodeForTree_DONE
0dd9da90 79440b77 mscorjit!emitter::emitNewInstrTiny+0xd, calling
mscorjit!emitter::emitAllocInstr
0dd9daa0 79440ba9 mscorjit!emitter::emitIns_R_R+0x33, calling mscorjit!
emitter::emitNewInstrTiny
0dd9dab4 7943a4bb mscorjit!Compiler::inst_RV_RV+0x26, calling mscorjit!
emitter::emitIns_R_R
0dd9dabc 791b4b81 mscorwks!MetaDataTracker::NoteAccess+0xc2, calling
mscorwks!_SEH_epilog

Willy Denoyette [MVP]
3/19/2007 8:42:49 PM

[quoted text, click to view]

esafran
3/19/2007 10:06:10 PM

On Mar 19, 9:42 pm, "Willy Denoyette [MVP]"
[quoted text, click to view]

Hi,

[quoted text, click to view]

You have to admit that it is very suspicious that with every run of
the process, when it hangs, I make a hang dump, and I see those
threads in the same place.
Besides, the run of these threads can be disabled by configuration,
and when the process launches with these threads disabled, the process
NEVER hangs:

Thread 57
ESP EIP
0x0dd9e520 0x7c90eb94 [FRAME: GCFrame]
0x0dd9efb4 0x7c90eb94 [FRAME: GCFrame]
0x0dd9f060 0x7c90eb94 [FRAME: PrestubMethodFrame] [DEFAULT] [hasThis]
Void DVTel.Logger.MotionDetection.MotionRenderer..ctor(String)
0x0dd9f070 0x03d6f40f [DEFAULT] [hasThis] Class
DVTel.Logger.Common.IMotionDetector
DVTel.Logger.MotionDetection.CapabilityMotionDetector.Start(Class
DVTel.API.Entities.Physical.IMotionDetectionProfileEntity,Class
DVTel.API.Entities.Physical.IRoiEntity)
at [+0x87] [+0x1e]
....

public MotionRenderer(string strName)
{
m_strname = strName;
ThreadName = string.Format("Motion Renderer {0}", Name);
m_rcCounter = new RefCounter();
m_rcCounter.FinalRelease += new EventHandler(FinalReleaseCallback);
m_rcCounter.FirstAddRef += new EventHandler(FirstAddRefCallback);
}

Thread 62
ESP EIP
0x1022e58c 0x7c90eb94 [FRAME: GCFrame]
0x1022f020 0x7c90eb94 [FRAME: GCFrame]
0x1022f0cc 0x7c90eb94 [FRAME: PrestubMethodFrame] [DEFAULT] [hasThis]
Void DVTel.Common.Streamer.SinkDecode_Video..ctor()
....

public SinkDecode_Video()
{
m_dehIFrameReady = new ThreadStart(IFrameReadyCallback);
m_dehDecodeFailure = new ThreadStart(OnDecodeFailure);
}

[quoted text, click to view]

1) if you look at the process using: "Process explorer NT", then
there is no activity in ANY of the threads
2) if you try to attach to the process with Visual Studio, you don't
see the IDE loading any of the assemblies in the "Output" window.
3) if you try to pause the process, after you attached to it, the IDE
tells you that it still didn't load the assemblies and that it
couldn't determine the location of each thread ( or a similar message)
4) we have a console window which logs everything out, and it too
stops displaying new logs

Moreover, you can see in the !dumpstack i sent, that Thread 57 is
waiting: "ZwWaitForSingleObject" and the stack from that point, down
to my code, is pretty big and it contains calls for loading assemlies.
I suspect that there is an OS lock in loading assemblies, I just can't
figure out where.
The only difference with these 2 threads from all other threads, is
that each thread of those 2, is running in an assembly which have a
reference to a Managed c++ assembly.
They are not located in the same assembly, and they do not reference
to the same Managed c++ assembly, but they do reference to a Managed c+
+ assembly.
maybe the OS lock is something to do with the Managed c++.

I hope the information i gave here will help.
Looking forward to hear from you.

Eyal Safran.

Ismo Salonen
3/20/2007 12:00:00 AM

--snip--

By any chance these are global objects initialized by from Dllmain ?
This looks like loader lock but I'm not sure, just a guess.

esafran
3/20/2007 5:39:50 AM

[quoted text, click to view]

I've made !locks command:

0:057> !locks

CritSec ntdll!LdrpLoaderLock+0 at 7C97C0D8
LockCount 2
RecursionCount 2
OwningThread 1b88
EntryCount 23
ContentionCount 23
*** Locked

CritSec mscorwks!CorMap::m_pCorMapCrst+0 at 793DF9B0
LockCount 0
RecursionCount 1
OwningThread 1b1c
EntryCount 0
ContentionCount 0
*** Locked

CritSec +1422d0 at 001422D0
LockCount 0
RecursionCount 1
OwningThread 1f50
EntryCount 4
ContentionCount 4
*** Locked

CritSec +5992ec4 at 05992EC4
LockCount 0
RecursionCount 1
OwningThread 1b1c
EntryCount 0
ContentionCount 0
*** Locked

CritSec +5a43244 at 05A43244
LockCount 0
RecursionCount 1
OwningThread 1b88
EntryCount 0
ContentionCount 0
*** Locked

Scanned 1971 critical sections

Thread 1b88 = Thread 62
Thread 1b1c = Thread 57

It appears to be a loader lock

Eyal Safran.

esafran
3/20/2007 7:42:53 AM

[quoted text, click to view]

Do you happen to know the link to this article of microsoft where they
state they won't fix it?

Eyal Safran.

esafran
3/20/2007 7:50:56 AM

[quoted text, click to view]

Do you know if by compiling my Managed C++ dlls in VC2005, but still
in .NET framework 1.1, will solve my loader lock problem?

Eyal Safran.

Ben Voigt
3/20/2007 8:00:48 AM

[quoted text, click to view]

Which version of C++.NET are your assemblies? Managed Extensions for C++
(VC2003) has known loader lock bugs that microsoft won't fix. Use C++/CLI
(VC2005) instead.

[quoted text, click to view]

Ben Voigt
3/20/2007 11:29:51 AM

[quoted text, click to view]

While all of .NET 1.1 is available in VC2005, I don't think you can target
the old metadata format. Therefore, the Framework 1.x runtime won't be able
to load VC2005 dlls even if they only use the 1.x BCL.

I'm looking for the Microsoft note that loader lock won't be fixed for
VC2003.

[quoted text, click to view]

Ben Voigt
3/20/2007 11:40:42 AM

[quoted text, click to view]

http://msdn2.microsoft.com/en-us/library/ms173266(vs.80).aspx

The solution is definitely stated as: use .NET 2.0, the .NET 1.1 runtime
(not actually a VS2003 vs VS2005 issue) causes random loader lock.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dv_vstechart/html/vcconmixeddllloadingproblem.asp

See also here, support for Managed Extensions for C++ will be removed
because C++/CLI replaces it.
http://msdn2.microsoft.com/en-us/library/b23b94s7(vs.80).aspx

[quoted text, click to view]

Willy Denoyette [MVP]
3/21/2007 12:00:00 AM

[quoted text, click to view]

No, this won't work, if you can't fix the "OS loader lock" issue by one of the resolutions
as mentioned in the article (Initialization of Mixed Assemblies ) pointed to by Ben, then
you are out of luck and you will have to move forward.

Willy.

esafran
3/21/2007 6:38:53 AM

On Mar 21, 11:16 am, "Willy Denoyette [MVP]"
[quoted text, click to view]

I've found out a scenario where it always hang, and so, i've found a
workaround to it.
The problem is in the order DLLs are being loaded.

Example:
I have a Managed class MBuffer which references to its Unmanaged class
CBuffer

I also have a Managed class MDecoder which references to its Unmanaged
class CDecoder and it references to the unmanaged CBuffer.

MBuffer -> CBuffer
MDecoder -> CDecoder
CDecoder -> CBuffer

now, if the process have loaded the DLLs in the following order, the
process will get stuck:

1) MBuffer
2) CBuffer
3) MDecoder
4) CDecoder

however, if the loading order of the DLLs would have been:
1) MDecoder
2) CDecoder
3) CBuffer
4) MBuffer

the process will never get stuck.

I really don't know why it acts like that.

If you have any idea on how to solve / debug that, that would be
great.
Currently I'm forcing synchronization between the 2 threads with a
static ManualResetEvent, so that the MBuffer DLL will not load untill
its CBuffer DLL will load.

Eyal Safran.

Got something to say? Click here to post a reply to this thread.

Don't see what you're looking for? Try a search.

View other messages about dotnet clr

dotnet clr : My entire process freezes. - Help.