|
|
You're in the
dotnet clr
group:Managed Memory Consumption - Service vs. Command Line EXE
dotnet clr:
[quoted text, click to view]
"Robert Reineri" <robertX123.reineri@fnbmarin.com> wrote in message news:u6to6vPcFHA.3560@TK2MSFTNGP10.phx.gbl... > Hello All, > > I have written an application to process a file full of XML. For each > "main element" in the xml file, the application parses it and writes the > resulting object to an Oracle database. Here is the problem. > > In an XML file with 20000 objects, running as a service, the silly thing > consums 500MB of memory. > > If I take the exact same code, and place it in a console app, it uses less > than 30 MB of memory. That is, I simply took what was in the service > OnStart() method, and placed it in Main() in the console app. I'm just > starting up a couple of threads: My guess is it's Server vs Workstation mode of the CLR. I'm not sure how you can do it with .NET 1.1 but with .NET 2.0 you can specify in your .config file which version of the clr (server or workstation). -- Regards Vincent Parrett VSoft Technologies http://www.finalbuilder.com Blogs : http://blogs.finalbuilder.com ---------------------------------------- Automate your Build Process with FinalBuilder
Hello All,
I have written an application to process a file full of XML. For each "main element" in the xml file, the application parses it and writes the resulting object to an Oracle database. Here is the problem. In an XML file with 20000 objects, running as a service, the silly thing consums 500MB of memory. If I take the exact same code, and place it in a console app, it uses less than 30 MB of memory. That is, I simply took what was in the service OnStart() method, and placed it in Main() in the console app. I'm just starting up a couple of threads: One thread reads and parses the XML (serially, not using the DOM), creates a lightweight object and populates a bunch of properties, and sticks the object in an MSMQ queue. The other thread reads these lightweight objects out of the queue, and serializes them to an Oracle database. I have been over and over the code, and I can see no place where I am using unmanaged objects, or holding on to references. This is most perplexing. As a service, it runs like crap. As a console app, it runs just fine. When I examine managed memory with Perfmon while running the service, the Gen2 heap size grows enormously. No objects are pinned. When I examine the same thing running as a console app, Gen2 never gets very big at all (maybe 35MB max). Any idea what is going on here? Is there something special I need to do in a service to manage memory. Please respond via newsgroup. When I post my email address, the spambots have a field day! Thanks Robert Reineri Sr. Business Systems Engineer Credit One Bank
No, not at all, your issue does not relate to this.
If you see a lot of objects ending in gen2, it's because you are: - holding live references to them, - or the GC cannot compact because there are pinned objects, or - in the case they are Finalizable objects, the finalizer cannot run for one or an other reason. It's hard to tell from the description you gave, it would be great if you could provide a sample that illustrates the issue. Willy. [quoted text, click to view] "Robert Reineri" <robertREMOVETHIS@reineri.org> wrote in message news:e7axsnacFHA.3864@TK2MSFTNGP10.phx.gbl... > Sooo... is there something I should be setting in the .config file for the > service??? > > Thanks > > Robert > > "Willy Denoyette [MVP]" <willy.denoyette@telenet.be> wrote in message > news:OG6GJrXcFHA.612@TK2MSFTNGP12.phx.gbl... >> Nope, it's not because your application is a "Service" that the CLR will >> switch to Server mode. >> Server and Workstation mode is selected through a Config parameter and >> requires multi-cpu boxes. >> >> Willy. >> >> "Vincent Parrett (VSoft Technologies)" <vincent@no-spam.finalbuilder.com> >> wrote in message news:uvN0WnVcFHA.3620@TK2MSFTNGP09.phx.gbl... >>> "Robert Reineri" <robertX123.reineri@fnbmarin.com> wrote in message >>> news:u6to6vPcFHA.3560@TK2MSFTNGP10.phx.gbl... >>>> Hello All, >>>> >>>> I have written an application to process a file full of XML. For each >>>> "main element" in the xml file, the application parses it and writes >>>> the resulting object to an Oracle database. Here is the problem. >>>> >>>> In an XML file with 20000 objects, running as a service, the silly >>>> thing consums 500MB of memory. >>>> >>>> If I take the exact same code, and place it in a console app, it uses >>>> less than 30 MB of memory. That is, I simply took what was in the >>>> service OnStart() method, and placed it in Main() in the console app. >>>> I'm just starting up a couple of threads: >>> >>> My guess is it's Server vs Workstation mode of the CLR. I'm not sure how >>> you can do it with .NET 1.1 but with .NET 2.0 you can specify in your >>> .config file which version of the clr (server or workstation). >>> >>> >>> -- >>> Regards >>> >>> Vincent Parrett >>> VSoft Technologies >>> http://www.finalbuilder.com >>> Blogs : http://blogs.finalbuilder.com >>> ---------------------------------------- >>> Automate your Build Process with FinalBuilder >>> >> >> > >
On Tue, 14 Jun 2005 09:11:10 -0700, "Robert Reineri"
[quoted text, click to view] <robertX123.reineri@fnbmarin.com> wrote: > Hello All, > > I have written an application to process a file full of XML. For each "main > element" in the xml file, the application parses it and writes the resulting > object to an Oracle database. Here is the problem. > > In an XML file with 20000 objects, running as a service, the silly thing > consums 500MB of memory. Sounds like a garbage collection issue to me: I've learned not to depend on the framework to keep it clean. You have two options (I think): force garbage collection regularly, or call ..Dispose() on everything that has such a method before it goes out of scope. That includes at times when something is made to point to another object of the same type, for example when you replace the image in a picturebox by another one: picturebox.Image.Dispose() picturebox.Image = NewImage
Sooo... is there something I should be setting in the .config file for the
service??? Thanks Robert [quoted text, click to view] "Willy Denoyette [MVP]" <willy.denoyette@telenet.be> wrote in message news:OG6GJrXcFHA.612@TK2MSFTNGP12.phx.gbl... > Nope, it's not because your application is a "Service" that the CLR will > switch to Server mode. > Server and Workstation mode is selected through a Config parameter and > requires multi-cpu boxes. > > Willy. > > "Vincent Parrett (VSoft Technologies)" <vincent@no-spam.finalbuilder.com> > wrote in message news:uvN0WnVcFHA.3620@TK2MSFTNGP09.phx.gbl... >> "Robert Reineri" <robertX123.reineri@fnbmarin.com> wrote in message >> news:u6to6vPcFHA.3560@TK2MSFTNGP10.phx.gbl... >>> Hello All, >>> >>> I have written an application to process a file full of XML. For each >>> "main element" in the xml file, the application parses it and writes the >>> resulting object to an Oracle database. Here is the problem. >>> >>> In an XML file with 20000 objects, running as a service, the silly thing >>> consums 500MB of memory. >>> >>> If I take the exact same code, and place it in a console app, it uses >>> less than 30 MB of memory. That is, I simply took what was in the >>> service OnStart() method, and placed it in Main() in the console app. >>> I'm just starting up a couple of threads: >> >> My guess is it's Server vs Workstation mode of the CLR. I'm not sure how >> you can do it with .NET 1.1 but with .NET 2.0 you can specify in your >> .config file which version of the clr (server or workstation). >> >> >> -- >> Regards >> >> Vincent Parrett >> VSoft Technologies >> http://www.finalbuilder.com >> Blogs : http://blogs.finalbuilder.com >> ---------------------------------------- >> Automate your Build Process with FinalBuilder >> > >
Robert,
the easiest way to figure out what objects are laying around for so long (and how big they are), is to use a profiler. MS has the free CLR Profiler which might help. There is also one called .Net Memory Profiler that i've had really good luck with (http://www.scitech.se/memprofiler/). bottom line, is that your just guessing unless you profile your app. That way you'll know for sure what the problem is [quoted text, click to view] "Robert Reineri" wrote:
> Hello All, > > I have written an application to process a file full of XML. For each "main > element" in the xml file, the application parses it and writes the resulting > object to an Oracle database. Here is the problem. > > In an XML file with 20000 objects, running as a service, the silly thing > consums 500MB of memory. > > If I take the exact same code, and place it in a console app, it uses less > than 30 MB of memory. That is, I simply took what was in the service > OnStart() method, and placed it in Main() in the console app. I'm just > starting up a couple of threads: > > One thread reads and parses the XML (serially, not using the DOM), creates a > lightweight object and populates a bunch of properties, and sticks the > object in an MSMQ queue. The other thread reads these lightweight objects > out of the queue, and serializes them to an Oracle database. > > I have been over and over the code, and I can see no place where I am using > unmanaged objects, or holding on to references. > > This is most perplexing. As a service, it runs like crap. As a console app, > it runs just fine. When I examine managed memory with Perfmon while running > the service, the Gen2 heap size grows enormously. No objects are pinned. > When I examine the same thing running as a console app, Gen2 never gets very > big at all (maybe 35MB max). > > Any idea what is going on here? Is there something special I need to do in a > service to manage memory. Please respond via newsgroup. When I post my email > address, the spambots have a field day! > > Thanks > > Robert Reineri > Sr. Business Systems Engineer > Credit One Bank > >
I agree, and will definetly profile it. I have the CLR profiler. What has me
stumped is why it works fine as a console app, and is a memory hog as a service. It implies there is nothing wrong with the code. What changes is the environment. I will try the profiler and see what happens. Thanks all Robert [quoted text, click to view] "john conwell" <johnconwell@discussions.microsoft.com> wrote in message news:17453EDA-ADE3-4A10-8DD8-BA1D22EB11EC@microsoft.com... > Robert, > the easiest way to figure out what objects are laying around for so long > (and how big they are), is to use a profiler. MS has the free CLR > Profiler > which might help. There is also one called .Net Memory Profiler that i've > had really good luck with (http://www.scitech.se/memprofiler/). > > bottom line, is that your just guessing unless you profile your app. That > way you'll know for sure what the problem is > > "Robert Reineri" wrote: > >> Hello All, >> >> I have written an application to process a file full of XML. For each >> "main >> element" in the xml file, the application parses it and writes the >> resulting >> object to an Oracle database. Here is the problem. >> >> In an XML file with 20000 objects, running as a service, the silly thing >> consums 500MB of memory. >> >> If I take the exact same code, and place it in a console app, it uses >> less >> than 30 MB of memory. That is, I simply took what was in the service >> OnStart() method, and placed it in Main() in the console app. I'm just >> starting up a couple of threads: >> >> One thread reads and parses the XML (serially, not using the DOM), >> creates a >> lightweight object and populates a bunch of properties, and sticks the >> object in an MSMQ queue. The other thread reads these lightweight objects >> out of the queue, and serializes them to an Oracle database. >> >> I have been over and over the code, and I can see no place where I am >> using >> unmanaged objects, or holding on to references. >> >> This is most perplexing. As a service, it runs like crap. As a console >> app, >> it runs just fine. When I examine managed memory with Perfmon while >> running >> the service, the Gen2 heap size grows enormously. No objects are pinned. >> When I examine the same thing running as a console app, Gen2 never gets >> very >> big at all (maybe 35MB max). >> >> Any idea what is going on here? Is there something special I need to do >> in a >> service to manage memory. Please respond via newsgroup. When I post my >> address, the spambots have a field day! >> >> Thanks >> >> Robert Reineri >> Sr. Business Systems Engineer >> Credit One Bank >> >> >>
Nope, it's not because your application is a "Service" that the CLR will
switch to Server mode. Server and Workstation mode is selected through a Config parameter and requires multi-cpu boxes. Willy. "Vincent Parrett (VSoft Technologies)" <vincent@no-spam.finalbuilder.com> [quoted text, click to view] wrote in message news:uvN0WnVcFHA.3620@TK2MSFTNGP09.phx.gbl... > "Robert Reineri" <robertX123.reineri@fnbmarin.com> wrote in message > news:u6to6vPcFHA.3560@TK2MSFTNGP10.phx.gbl... >> Hello All, >> >> I have written an application to process a file full of XML. For each >> "main element" in the xml file, the application parses it and writes the >> resulting object to an Oracle database. Here is the problem. >> >> In an XML file with 20000 objects, running as a service, the silly thing >> consums 500MB of memory. >> >> If I take the exact same code, and place it in a console app, it uses >> less than 30 MB of memory. That is, I simply took what was in the service >> OnStart() method, and placed it in Main() in the console app. I'm just >> starting up a couple of threads: > > My guess is it's Server vs Workstation mode of the CLR. I'm not sure how > you can do it with .NET 1.1 but with .NET 2.0 you can specify in your > .config file which version of the clr (server or workstation). > > > -- > Regards > > Vincent Parrett > VSoft Technologies > http://www.finalbuilder.com > Blogs : http://blogs.finalbuilder.com > ---------------------------------------- > Automate your Build Process with FinalBuilder >
Bad suggestion, which doesn't in no way explain why it runs from the console
application, the GC has no knowledge about what kind of application it is - service or console , calling GC.Collect yourself only disturbs the normal functioning of the GC. If a Gen2 increases to 500MB the GC has run several thousand times already, just watch the GC performance counters. Willy. [quoted text, click to view] "Lucvdv" <replace_name@null.net> wrote in message news:7di0b1hkedpj3tpaes3b77snadumlrmuan@4ax.com... > On Tue, 14 Jun 2005 09:11:10 -0700, "Robert Reineri" > <robertX123.reineri@fnbmarin.com> wrote: > >> Hello All, >> >> I have written an application to process a file full of XML. For each >> "main >> element" in the xml file, the application parses it and writes the >> resulting >> object to an Oracle database. Here is the problem. >> >> In an XML file with 20000 objects, running as a service, the silly thing >> consums 500MB of memory. > > Sounds like a garbage collection issue to me: I've learned not to depend > on > the framework to keep it clean. > > > You have two options (I think): force garbage collection regularly, or > call > .Dispose() on everything that has such a method before it goes out of > scope. > > That includes at times when something is made to point to another object > of > the same type, for example when you replace the image in a picturebox by > another one: > > picturebox.Image.Dispose() > picturebox.Image = NewImage >
That's not what the GC is used for, the GC manages the "managed heap", that
is memory used to store the "managed objects", that's all. The GC doesn't know anything about unmanaged memory nor about unmanaged resources like handles and files, so it's up to you to clean up unmanaged stuff by disposing objects that implement IDisposable. The GC cannot compact the heap if there are a lot of pinned objects, PInvoke interop and COM interop are the manifest causes of GC collection prevention, so watch out for dangling COM object references and long running PInvoke calls. If ever you have notices memory increases to 500MB it's not because the GC doesn't run, again do yourself a favor and look at the GC memory counters using perfmon, you will see that the GC runs more often than you think, but the GC cannot free objects that are still referenced, forcing a GC Collect wont help anyway. Another point to consider is that the GC doesn't compact memory that holds large objects (>85Kb), it's possible that this "Large Object Heap" gets so badly fragmented that no memory can be returned to the CLR nor the OS, so watch out for large objects. Willy. [quoted text, click to view] "Lucvdv" <replace_name@null.net> wrote in message news:3905b1ha88gvm6f6im70jrqgin31mi2t3r@4ax.com... > On Wed, 15 Jun 2005 18:45:46 +0200, "Willy Denoyette [MVP]" > <willy.denoyette@telenet.be> wrote: > >> Bad suggestion, which doesn't in no way explain why it runs from the >> console >> application, the GC has no knowledge about what kind of application it >> is - >> service or console , calling GC.Collect yourself only disturbs the normal >> functioning of the GC. >> >> If a Gen2 increases to 500MB the GC has run several thousand times >> already, >> just watch the GC performance counters. > > The more I use .Net, the more I get convinced that GC doesn't kick in even > remotely soon enough. > > By relying on automatic garbage collection, I've seen a standard VB.Net > app > take up 600-700 MB on a 1GB RAM system when it didn't need more than a few > tens of MB at the time. > > That application had loaded a lot of small images, most of which had gone > out of scope. They remained in memory, and garbage collection only kicked > in after the app had eaten up over 600 MB. > > Two solutions solved it: either force garbage collection regularly (not > the > best way, I agree - prevention is better than fixing symptoms), or call > Dispose() on each image before it went out of scope. > I implemented the latter. > > > Sometimes it's even worse than just keeping memory occupied. The CLR also > keeps files open unnecessarily, as a simple example illustrates: put this > code in the load event of a form with a picturebox (both jpeg files must > exist, of course). > > PictureBox1.Image = Image.FromFile("img1.jpg") > PictureBox1.Image = Image.FromFile("img2.jpg") > > Start the app, then try to delete file "img1.jpg". It won't work before > the app is stopped, because the CLR still has the file open. > > To solve it either include a line "PictureBox1.Image.Dispose()" before > loading the second image, or force garbage collection after loading it. >
On Wed, 15 Jun 2005 18:45:46 +0200, "Willy Denoyette [MVP]"
[quoted text, click to view] <willy.denoyette@telenet.be> wrote: > Bad suggestion, which doesn't in no way explain why it runs from the console > application, the GC has no knowledge about what kind of application it is - > service or console , calling GC.Collect yourself only disturbs the normal > functioning of the GC. > > If a Gen2 increases to 500MB the GC has run several thousand times already, > just watch the GC performance counters. The more I use .Net, the more I get convinced that GC doesn't kick in even remotely soon enough. By relying on automatic garbage collection, I've seen a standard VB.Net app take up 600-700 MB on a 1GB RAM system when it didn't need more than a few tens of MB at the time. That application had loaded a lot of small images, most of which had gone out of scope. They remained in memory, and garbage collection only kicked in after the app had eaten up over 600 MB. Two solutions solved it: either force garbage collection regularly (not the best way, I agree - prevention is better than fixing symptoms), or call Dispose() on each image before it went out of scope. I implemented the latter. Sometimes it's even worse than just keeping memory occupied. The CLR also keeps files open unnecessarily, as a simple example illustrates: put this code in the load event of a form with a picturebox (both jpeg files must exist, of course). PictureBox1.Image = Image.FromFile("img1.jpg") PictureBox1.Image = Image.FromFile("img2.jpg") Start the app, then try to delete file "img1.jpg". It won't work before the app is stopped, because the CLR still has the file open. To solve it either include a line "PictureBox1.Image.Dispose()" before loading the second image, or force garbage collection after loading it.
[quoted text, click to view]
Willy Denoyette [MVP] wrote: > If ever you have notices memory increases to 500MB it's not because the GC > doesn't run, Then why did the problem go away by forcing GC? That was the first solution I tried, I started calling Dispose() on images that were about to go out of scope only later. I have looked at the GC counters yesterday while writing that reply. VS.Net had been open constantly for over 2 hours, a second instance had been open with another application for about half an hour, and a program was running in the debugger for a couple of minutes. There were 4 instances of the GC counters at that moment. At least two were at all zeroes, the _Global_ instance was at 1 for each level (I don't remember checking the 4th instance).
[quoted text, click to view] "Lucvdv" <replace_name@null.net> wrote in message news:snc7b1thifqfb8558bp9uk9uojmdss45ug@4ax.com... > Willy Denoyette [MVP] wrote: > >> If ever you have notices memory increases to 500MB it's not because the >> GC >> doesn't run, > > Then why did the problem go away by forcing GC? > That was the first solution I tried, I started calling Dispose() on > images that were about to go out of scope only later. > > > I have looked at the GC counters yesterday while writing that reply. > > VS.Net had been open constantly for over 2 hours, a second instance had > been open with another application for about half an hour, and a program > was running in the debugger for a couple of minutes. > > There were 4 instances of the GC counters at that moment. At least two > were at all zeroes, the _Global_ instance was at 1 for each level (I > don't remember checking the 4th instance). > Run your code in release mode out of the debugger and watch the perf counters for gen0, 1, 2 (size of and number of collection runs). It's not possible they are at zero, when the gen0 size exceeds a certain threshold the GC kicks in, this threshold is lower than 512Kb in general on v1.1 (it's somewhat higher on v2.0 where the GC is less aggressive), so it should kick in each time the threshold is exceeded possibly move the remaining live objects to gen1. The same thing is done for gen1 (the threshold being larger ), surviving objects are move to gen2 where they stay as long as they live. The story for large objects (>85kb) is different though, they are only collected when the GC does a full collect (gen2 collect) or a forced GC.Collect(). Note that this heap, as I told you before, is never compacted!, this leads to memory overflows when the LOH gets badly fragmented. Willy.
I was about to throw in a comment about release/debug. The GC is very
different (almost non-existent) when you run in debug mode. Release builds are essential. -- Phil Wilson [MVP Windows Installer] ---- [quoted text, click to view] "Willy Denoyette [MVP]" <willy.denoyette@telenet.be> wrote in message news:uF271R%23cFHA.2384@TK2MSFTNGP10.phx.gbl... > > "Lucvdv" <replace_name@null.net> wrote in message > news:snc7b1thifqfb8558bp9uk9uojmdss45ug@4ax.com... >> Willy Denoyette [MVP] wrote: >> >>> If ever you have notices memory increases to 500MB it's not because the >>> GC >>> doesn't run, >> >> Then why did the problem go away by forcing GC? >> That was the first solution I tried, I started calling Dispose() on >> images that were about to go out of scope only later. >> >> >> I have looked at the GC counters yesterday while writing that reply. >> >> VS.Net had been open constantly for over 2 hours, a second instance had >> been open with another application for about half an hour, and a program >> was running in the debugger for a couple of minutes. >> >> There were 4 instances of the GC counters at that moment. At least two >> were at all zeroes, the _Global_ instance was at 1 for each level (I >> don't remember checking the 4th instance). >> > > Run your code in release mode out of the debugger and watch the perf > counters for gen0, 1, 2 (size of and number of collection runs). It's not > possible they are at zero, when the gen0 size exceeds a certain threshold > the GC kicks in, this threshold is lower than 512Kb in general on v1.1 > (it's somewhat higher on v2.0 where the GC is less aggressive), so it > should kick in each time the threshold is exceeded possibly move the > remaining live objects to gen1. The same thing is done for gen1 (the > threshold being larger ), surviving objects are move to gen2 where they > stay as long as they live. > The story for large objects (>85kb) is different though, they are only > collected when the GC does a full collect (gen2 collect) or a forced > GC.Collect(). Note that this heap, as I told you before, is never > compacted!, this leads to memory overflows when the LOH gets badly > fragmented. > > Willy. > > > > >
On Mon, 20 Jun 2005 15:05:33 -0700, "Phil Wilson"
[quoted text, click to view] <Phil.Wilson@unisys.spamcom> wrote: > I was about to throw in a comment about release/debug. The GC is very > different (almost non-existent) when you run in debug mode. Release builds > are essential. That would explain all or most of the symptoms that lead me to say GC doesn't kick in nearly often enough. On my own system, it's virtually always debug builds.
Don't see what you're looking for? Try a search.
|
June 2003
July 2003
August 2003
September 2003
October 2003
November 2003
December 2003
January 2004
February 2004
March 2004
April 2004
May 2004
June 2004
July 2004
August 2004
September 2004
October 2004
November 2004
December 2004
January 2005
February 2005
March 2005
April 2005
May 2005
June 2005
July 2005
August 2005
September 2005
October 2005
November 2005
December 2005
January 2006
February 2006
March 2006
April 2006
May 2006
June 2006
July 2006
August 2006
September 2006
October 2006
November 2006
December 2006
January 2007
February 2007
March 2007
April 2007
May 2007
June 2007
July 2007
August 2007
September 2007
October 2007
November 2007
December 2007
January 2008
February 2008
March 2008
April 2008
May 2008
June 2008
| home · blog · groups | Questions? Comments? Contact the dn |