Out of memory -- dotnet clr

Marek Cvrk
9/13/2005 9:41:53 AM

Dataset is simple, only one table with about 20 columns. No data are in
dataset during construction.

We run server as service in debug build. Is there some difference between GC
mem. management in debug/release build?

We have problem with big gen 2 heap size, a lot of memory goes into and then
is collected. But I do not see any direct relation between these problems.

There is a lot of available memory on machine, but we do not have data from
all memory counters. It seems that less than 2GB is adressed.

Any other ideas?

Thanks.
Marek

[quoted text, click to view]

Willy Denoyette [MVP]
9/13/2005 7:56:31 PM

Large Gen2 size might be an indication that you have a lot of aging objects,
but as you correctly pointed out the GC is less agressive when running debug
builds, anyway you have to check your object usage pattern running a memory
profiler.

Willy.

[quoted text, click to view]

Willy Denoyette [MVP]
9/13/2005 7:59:11 PM

These aren't considered at all, they are only hints for the memory manager,
the process workingset might grow way above MaxWorkingSet and is never the
cause of OOM exceptions.

Willy.

[quoted text, click to view]

Willy Denoyette [MVP]
9/14/2005 12:00:00 AM

[quoted text, click to view]

Please start with the suggestions given in this thread, check your memory
allocation pattern, you are consume too much memory probably because you are
not releasing unmanaged resources (use Dispose() when needed), or your
allocation pattern consists of many large objects (>85Kb) allocated in many
different sizes which makes that the large object heap gets badly
fragmented. Only way to know what happens ... is to MEASURE, use memory
profilers, the performance monitor and it's CLR memory counters a low level
debugger whatever you see fit, but measure.

Willy.

Marek Cvrk
9/14/2005 12:48:05 AM

I have some more symptoms, remoting calls are blocked for a long time on
server side, so clients has no response. Should it be caused by GC large
collection? I have read that all working threads are stopped during large
collection.

Marek.

[quoted text, click to view]

Marek Cvrk
9/14/2005 1:39:03 AM

And finally, there is 3GB of physical memory on machine and when problem came
on memory available counter showed 1,3GB. That is all information I have.

Thanks for helping.

Marek.

[quoted text, click to view]

David Browne
9/14/2005 8:12:47 AM

[quoted text, click to view]

Right. On x86 you can only address 2GB of memory, and the CLR heaps may
well not be able to grow past 1.3GB, since growing may require a larger
contiguous address space than is available.

Your problem is that you really are out of memory. Investigate why you are
using 1.3GB of memory. This is an obscene amount of memory to be using, and
usually indicates that you have a problem in your application. A .NET
memory profiler can tell you what is using all that memory, and why it
hasn't been garbage collected. This is usually because you are
inadvertently keeping a reference to dead objects.

David

Willy Denoyette [MVP]
9/14/2005 10:27:54 AM

That's right, when GC occurs all managed threads are suspended, but in
general this should not take a long time.
But again, these are all things you can measure don't try to guess, the CLR
memory counters are just there for this. Take a look at the % time spent in
the GC.

Willy.

[quoted text, click to view]

Willy Denoyette [MVP]
9/19/2005 12:00:00 AM

[quoted text, click to view]

There is not anough info in the dumps to pin-point the problem, however here
is what I see in the dump:

First you start with two large free memory area's (plus some smaller area's)
1. 11022000 -> 51a70000 = ~1GB
2. 51af0000 -> 61e50000 = ~ 260MB

The reason for this is (partly):
ModLoad: 51a70000 51af0000
C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\diasymreader.dll
Note that this DLL is badly mapped, it's a debugging helper DLL, are you
sure this is a release build?

You are running the server GC on a dual proc, that means you have two GC
heaps (one per processor), however, I don't see where they are allocated nor
what size they have.
Anyway, 959,716,812 is taken by the generational heaps (total), from which
953,434,664 is free, I don't see any large objects in the dump, so I'm not
sure your problem is related to the regular heap or the LOH.
The last part of the dump shows the fragmentation of the GC heap (the
953,434,664 free part), this is an indication that you are allocating
pinned objects combined with an high allocation rate.
Here is what I think is happening; when you run the Server GC , chances are
that your socket related code runs on one and the same single CPU (the CPU
handling the socket IO and APC stuff, the OS likes to keep this on the same
CPU if possible), so socket data will be stored on one and the same GC heap.
Socket data is pinned for the duration of the call to the (unmanaged)
Winsock library and if you don't take care the buffer might stay pinned for
a much longer time. When you run the WKS GC, you have one single heap shared
by both CPU's, the amount of fragmentation will probably be the same, but
the GC is more agressive, so he collects more often , the result being a
larger heap and more free space (larger free block).
Conclusion: check your socket data allocation and pinning, check the # of
pinned objects using perfmon and try to minimize them.

Willy.

Josef Dvorak
9/19/2005 5:09:01 AM

Hi,
I'm Marek Cvrk's colleague and we made some first measurement.
At server side we used "server settings" for Garbage Collector (element
<gcServer enabled="true"/> in config file - that sets to server type of GC)
and the server runnig on Windows 2000 Server OS and has 3GB of RAM.

With these settings service get OutOfMemory exception after some time of
runnig. The counter Private Bytes shows value abou 1,2GB.
So we try to delete element <gcServer> from config file (that sets to
workstation type of GC) and it works. But why? Any idea? (The counter
Private Bytes shows value about 600MB).

We also made memory dump of server with server GC (when Private Bytes shows
about 1,2GB; we use WinDbg) and it's look like memory fragmentation. (You can
find it on www.josef-dvorak.webzdarma.cz/dump2003_20050919.txt). At this dump
you can see (almost at the end) that there is about 1GB of free memory. But
in some cases it failed on OutOfMemoryException and in other cases memory is
released (and it falls to 400MB (starting memory consuption of "clear"
service) and proces of memory consumption will start again...). Any idea how
to prevent fragmentation?
There is no fragmentation with "workstation" GC.

We also read discussion about GC problems on W2k server with 2GB RAM or
more...
but nobody (at that discussion) confirmed that.

Thanks
Josef Dvorak

Josef Dvorak
9/19/2005 6:58:09 AM

Hi
if it's start, it has about 200MB. There were cache (a lot) which grows...
From heap dump you can see, that there was about 1G free memory, but
fragmented...

If we interpolated Virual Bytes counter from "workstation" GC, we get
"constant" line. But when we interpolated same from "server" GC, we get slow
increasing line.

We also using SSPI (example from MSDN) via .NET remoting and there were same
memory leaks, (STAThread vs. MTAThread attribute) but it looks that was
solved...

I think the value about 600MB of virtual memory is acceptable, but i'm
worried about "free" memory in heap dump.

David Browne
9/19/2005 8:02:59 AM

[quoted text, click to view]

There is always some fragmentation. Server GC uses one heap per processor.
More heaps, more potential for fragmentation.

So I would expect you to be able to access more of your limited address
space with Workstation GC.

But even 600MB is quite a bit of memory, perhaps a memory profiler could
help identify what is using all that memory.

David

Josef Dvorak
9/26/2005 7:28:04 AM

Hi Willy

we are using .NET Remoting with SSPI (example from MSDN) and only "our" (=
we know about them) pinned objects are in SSPI.
We try analyze heap's dumps from WinDbg (specially fragmented parts).

All object which are pinned are this (dump from WinDbg):
HANDLE(Pinned):9d2354:Root:0x23f39b7c(System.Byte[])
HANDLE(Strong):9d2b04:Root:0xc2eef8(System.Net.Sockets.OverlappedAsyncResult)->0x4d3dc20(System.AsyncCallback)->0x4d3dbc0(System.Runtime.Remoting.Channels.Tcp.TcpServerSocketHandler)->0x23f39b7c(System.Byte[])

It's always "after" free space in heap. Any idea?

PS:
MSDN example of SSPI is:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndotnet/html/remsspi.asp
Dump from WinDbg is: http://www.josef-dvorak.webzdarma.cz/dumpXP_20050926.txt

Thanks Josef

[quoted text, click to view]

s_mondrup
10/12/2005 12:23:04 AM

I have the very same problem:

[quoted text, click to view]

I also use .NET remoting with SSPI and the problem seems to be worse when
running server-gc. But we still see massive heap-fragmentation with
workstation-gc and the performance drops to 1/4.

Have you found a solution to your problem?

Kind regards SÃ¸ren Mondrup

Josef Dvorak
10/12/2005 12:44:06 AM

Hi

we solved this issue by implementing GenuineChannel
(www.genuinechannels.com) into our project.

But we also solving this issue with microsoft (.Net Remoting + SSPI example
from MSDN) and i think this solution takes long time.

Kind regards
Josef Dvorak

[quoted text, click to view]

s_mondrup
10/12/2005 12:59:19 AM

Thanks

I also have Microsoft looking at the problem and they haven't come up with
any suggestions yet. Maybe the two support teams could join forces?

I'll check out GenuineChannel right away.

:) SÃ¸ren

Got something to say? Click here to post a reply to this thread.

Don't see what you're looking for? Try a search.

View other messages about dotnet clr

dotnet clr : Out of memory