Garbage Collection/OutOfMemory Exception 2 -- dotnet clr

ronald NO[at]SPAM calleva.com
1/8/2005 1:50:45 PM

Follow up to previous 16 message thread. Real world example of how
Garbage collector completely messes up machines and fails
comprehensively to collect Garbage with out prompting.

The Garbage collector is designed and optimised for Client based
applications where there are regular pauses in usage. It fails with
applications that either use large amounts of memory or run
continuously.

One suggestion a flag on the Garbage collection to set it collect
everything as soon as it becomes available. This would result in much
less memory fragmentation and allow intensive programs to run.

I am writing an indexing prototype in dot Net.

It uses a large hashtable (2.5 million items with more being added)
for lookups and sorted lists for results. The sorted lists are being
created and destroyed along with word arrays.

Stage 1 relying on Garbage collection with 500meg physical memory and
2 gig virtual single proc

System froze half way through indexing first category in first large
website.

Stage 2 Put in GC.Collect at a few points

Got to about 200 sites before freezing up.
Could get system to break, and tried a GC.Collect in immediate window.
Took about 10 minutes to return, was able to make a cup of coffee,
whilst the memory usage increased from 280 to around 380 before
dropping back to around 280. Dev enviroment then crashed.

Stage 3 got machine with 1gig ram dual proc no swap file configured.
Crashed with standard .net out of memory error at 6,000 sites
using Just GC.Collect in every loop

Stage 4 1 gig physical 4 gig virtual

using GC.Collect() and GC.WaitForFinalisers()

Ran throught to 28,000 sites before going to infinate cpu usage and
stopping processing.

Had to kill app and development enviroment. After log off and logon
still unable to run app, needed to reboot to clean up memory.

This version locked up at less than 500meg of memory usage.

The Large Object Heap size was 100meg and the gen 2 heap 341meg

See the following which is the output from the perf monitor as a
webpage. Unfortunately it is not in a readable format without saving
it to a text file and viewing in a browser.

It has gen 0,1,2 collections, heap size, induced gc etc etc.

Any ideas for how to proceed much appreaciated.

mail ronald . duncan at software . plc . uk

<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html;" />
<META NAME="GENERATOR" Content="Microsoft System Monitor" />
</HEAD>
<BODY>
<OBJECT ID="DISystemMonitor1" WIDTH="100%" HEIGHT="100%"
CLASSID="CLSID:C4D2D8E0-D1DD-11CE-940F-008029004347">
<PARAM NAME="_Version" VALUE="393219"/>
<PARAM NAME="_ExtentX" VALUE="15610"/>
<PARAM NAME="_ExtentY" VALUE="12859"/>
<PARAM NAME="DisplayType" VALUE="3"/>
<PARAM NAME="ReportValueType" VALUE="0"/>
<PARAM NAME="MaximumScale" VALUE="100"/>
<PARAM NAME="MinimumScale" VALUE="0"/>
<PARAM NAME="ShowLegend" VALUE="-1"/>
<PARAM NAME="ShowToolbar" VALUE="-1"/>
<PARAM NAME="ShowScaleLabels" VALUE="-1"/>
<PARAM NAME="ShowHorizontalGrid" VALUE="0"/>
<PARAM NAME="ShowVerticalGrid" VALUE="0"/>
<PARAM NAME="ShowValueBar" VALUE="-1"/>
<PARAM NAME="ManualUpdate" VALUE="0"/>
<PARAM NAME="Highlight" VALUE="0"/>
<PARAM NAME="ReadOnly" VALUE="0"/>
<PARAM NAME="MonitorDuplicateInstances" VALUE="-1"/>
<PARAM NAME="UpdateInterval" VALUE="1"/>
<PARAM NAME="DisplayFilter" VALUE="1"/>
<PARAM NAME="BackColorCtl" VALUE="-2147483633"/>
<PARAM NAME="ForeColor" VALUE="-1"/>
<PARAM NAME="BackColor" VALUE="-2147483633"/>
<PARAM NAME="GridColor" VALUE="8421504"/>
<PARAM NAME="TimeBarColor" VALUE="255"/>
<PARAM NAME="Appearance" VALUE="-1"/>
<PARAM NAME="BorderStyle" VALUE="0"/>
<PARAM NAME="NextCounterColor" VALUE="0"/>
<PARAM NAME="NextCounterWidth" VALUE="3"/>
<PARAM NAME="NextCounterLineStyle" VALUE="0"/>
<PARAM NAME="GraphTitle" VALUE=""/>
<PARAM NAME="YAxisLabel" VALUE=""/>
<PARAM NAME="DataSourceType" VALUE="1"/>
<PARAM NAME="SqlDsnName" VALUE=""/>
<PARAM NAME="SqlLogSetName" VALUE=""/>
<PARAM NAME="LogFileCount" VALUE="0"/>
<PARAM NAME="AmbientFont" VALUE="-1"/>
<PARAM NAME="LegendColumnWidths"
VALUE="-11.25 -11.25 -20.25 -15.25 -13 -13 -19"/>
<PARAM NAME="LegendSortDirection" VALUE="0"/>
<PARAM NAME="LegendSortColumn" VALUE="2097272"/>
<PARAM NAME="CounterCount" VALUE="48"/>
<PARAM NAME="MaximumSamples" VALUE="100"/>
<PARAM NAME="SampleCount" VALUE="100"/>
<PARAM NAME="SampleIndex" VALUE="92"/>
<PARAM NAME="StepNumber" VALUE="93"/>
<PARAM NAME="Counter00001.Path" VALUE="\\SEARCHINDEX\.NET CLR
Memory(_Global_)\# Gen 0 Collections"/>
<PARAM NAME="Counter00001.Color" VALUE="16711935"/>
<PARAM NAME="Counter00001.Width" VALUE="1"/>
<PARAM NAME="Counter00001.LineStyle" VALUE="0"/>
<PARAM NAME="Counter00001.ScaleFactor" VALUE="-1"/>
<PARAM NAME="Counter00001.Minimum" VALUE="100777"/>
<PARAM NAME="Counter00001.Maximum" VALUE="100777"/>
<PARAM NAME="Counter00001.Average" VALUE="100777"/>
<PARAM NAME="Counter00001.StatisticStatus" VALUE="0"/>
<PARAM NAME="Counter00001.Data"
VALUE="100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 1
0777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 100777 -1"/>
<PARAM NAME="Counter00002.Path" VALUE="\\SEARCHINDEX\.NET CLR
Memory(_Global_)\# Gen 1 Collections"/>
<PARAM NAME="Counter00002.Color" VALUE="4210752"/>
<PARAM NAME="Counter00002.Width" VALUE="1"/>
<PARAM NAME="Counter00002.LineStyle" VALUE="0"/>
<PARAM NAME="Counter00002.ScaleFactor" VALUE="-1"/>
<PARAM NAME="Counter00002.Minimum" VALUE="94191"/>
<PARAM NAME="Counter00002.Maximum" VALUE="94191"/>
<PARAM NAME="Counter00002.Average" VALUE="94191"/>
<PARAM NAME="Counter00002.StatisticStatus" VALUE="0"/>
<PARAM NAME="Counter00002.Data"
VALUE="94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 941
1 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 94191 -1"/>

Joakim Karlsson
1/9/2005 8:51:23 PM

[quoted text, click to view]

Actually there are two versions of the execution engine, one workstation
version (mscorwks.dll) and one server version (mscorsvr.dll).

[quoted text, click to view]

This suggests that the GC waits for the application's threads to become
idle before performing a collect. This is not the case. The GC kicks in
as soon as an allocation makes generation 0 exceed it's quota. This
causes all threads to be suspended or hijacked and a collect occur.

[quoted text, click to view]

This is impossible! Garbage collection *is* the process of determining
what memory areas are available and collecting those that are. The only
way to accomplish this is to make the developer explicitly tell the
runtime when it is done with an object. But wait... that's just what
we've done in C++ all along! I sure don't want to get back to that!

[quoted text, click to view]

<snip>

[quoted text, click to view]

The fact that you have objects that are generation 2 means that these
objects has survived 2 garbage collections. This means that they are
referenced by something in your code. This suggests that you actually
are still referencing these resources. Are you sure that you don't keep
references to your lists or from your lists to your objects?

dotnet clr : Garbage Collection/OutOfMemory Exception 2