|
|
You're in the
dotnet clr
group:Garbage Collection - Strange Results.
dotnet clr:
msdn as well other sites on IDisposable and heap generations. I am still a bit confused as to why the application I have written is acting in such "weird/awkward" manner. Basic outline of the application goes: Multi threaded socket application using Asynchronous writes/reads/connects via TCP Sockets. I have written an application that starts 100 threads and attaches 10 sockets from a peer to each of these threads; thus 1000 active sockets open. Now. the application runs fine; I calculate using ANTS profiler or any other profiling utility and it tells me that I have approximately 30 MB of Memory used. I call GC.GetTotalMemory(false) or even (true) and it gives me approximately the same values depending what stage of processing each thread is at. All threads are doing the same thing by the way. When I look in Windows Performance Monitor and under .NET Memory for that particular application the memory seems to be quite high. Several Hundred MEGABYTES. The application runs over time.. slowly but surely increasing in memory. as the threads complete their tasks with the sockets it closes them. and Disposes each and every object. I explicitly set each object to null as well its children.. and have inherited IDisposable on almost all objects. What I don't understand is why the memory keeps increasing, while the application keeps reporting only 30MB of usage... the memory keeps increasing and increasing until the system runs out of Physical Memory and causes OutOfMemoryExpections.... its quite strange. I am not doing anything within the application that would create anything larger then it should. However I do thousands of XmlSerializations using a single MemoryStream for each socket that is created; there are thousands of Asychronous callbacks that are happening. My Generation 0 and 1 heaps seem to be doing just fine.. but 2nd gen heap seems to be at hundreds of megs; its very strange to see this. I assumed that the .NET GC will keep on expanding the space required until it is ready for a garbage collection.. would it be because of the magnitute of ArrayLists that I am using ? or perhaps there are scattered memory segments. throughout the Heaps ? is the GC having a difficult time collecting all the resources? Any help on why this is occuring would be appreciated. I have tried my best to clean up all the resources.. but I figure that most of the objects created should automatically get cleaned up after it is out of the final scope that it is passed to.. for example. once a command comes in via the Async Read Callback, I process it and pass it to the appropriate thread DataReceive method. I not only dispose the object within CallBack but I also dispose the reference that was passed again just incase.. doesn't seem to be doing anything .. and what is even more strange is why is .NET or all Profilers such as ANTS .NET profiler report that I am only using 30MB of memory ? when in Task Manager or Performance Monitor I see hundreds of megs like 600+.
[quoted text, click to view] "CodeGuru" <CodeGuru@discussions.microsoft.com> wrote in message news:602B1F08-9484-420C-B157-2652E54E046B@microsoft.com... > Althought I have read hundreds if not more pages on > msdn as well other sites on IDisposable and heap > generations. I am still a bit confused as to why > the application I have written is acting in such > "weird/awkward" manner. > > Basic outline of the application goes: > > Multi threaded socket application using Asynchronous > writes/reads/connects via TCP Sockets. > > I have written an application that starts 100 threads > and attaches 10 sockets from a peer to each of these > threads; thus 1000 active sockets open. > > Now. the application runs fine; I calculate using ANTS profiler > or any other profiling utility and it tells me that I have approximately > 30 MB of Memory used. > > I call GC.GetTotalMemory(false) or even (true) and it gives me > approximately > the same values depending what stage of processing each thread > is at. > > All threads are doing the same thing by the way. > > When I look in Windows Performance Monitor and under .NET Memory > for that particular application the memory seems to be quite high. > > Several Hundred MEGABYTES. > > The application runs over time.. slowly but surely increasing in memory. > as the threads complete their tasks with the sockets it closes them. > and Disposes each and every object. I explicitly set each object to null > as well its children.. and have inherited IDisposable on almost all > objects. > > What I don't understand is why the memory keeps increasing, > while the application keeps reporting only 30MB of usage... the memory > keeps increasing and increasing until the system runs out of Physical > Memory > and causes OutOfMemoryExpections.... its quite strange. I am not doing > anything within the application that would create anything larger then it > should. > > However I do thousands of XmlSerializations using a single MemoryStream > for each socket that is created; there are thousands of Asychronous > callbacks > that are happening. > > My Generation 0 and 1 heaps seem to be doing just fine.. but 2nd gen heap > seems to be at hundreds of megs; its very strange to see this. > > I assumed that the .NET GC will keep on expanding the space required until > it is ready for a garbage collection.. would it be because of the > magnitute of > ArrayLists that I am using ? or perhaps there are scattered memory > segments. > throughout the Heaps ? is the GC having a difficult time collecting all > the > resources? > > Any help on why this is occuring would be appreciated. > > I have tried my best to clean up all the resources.. but I figure that > most of the objects created should automatically get cleaned up > after it is out of the final scope that it is passed to.. for example. > once a command comes in via the Async Read Callback, I process it > and pass it to the appropriate thread DataReceive method. > I not only dispose the object within CallBack but I also dispose > the reference that was passed again just incase.. > > doesn't seem to be doing anything .. and what is even more strange > is why is .NET or all Profilers such as ANTS .NET profiler report > that I am only using 30MB of memory ? when in Task Manager > or Performance Monitor I see hundreds of megs like 600+. > > Thank you and great weekend to all. 1. Don't use taskman, use perfmon and the "CLR Memory" counters Gen0, 1 and 2 size to watch managed memory consumption. 2. Take a look at the # of pinned objects using perfmon (CLR memory counters). 3. Use perfmon to watch your "process" privates bytes. If the the Gen 2 counter indicates a steady growth and the number of pinned objects is rather high (> 100) chances are that pinned objects are aging into Gen2 and as such preventing compactation of the heap. If that's the case you are probably holding references to socket (read/write) buffers for the lifetime of the socket. I would also suggest you to reduce the number of threads, 100 threads is only a waste of resources (both memory and CPU) when using asynchronous sockets. Willy.
[quoted text, click to view]
CodeGuru wrote: Just a guess if you're using asynchronous calls : Do you call EndInvoke for <snip> > for example. > once a command comes in via the Async Read Callback, I process it > and pass it to the appropriate thread DataReceive method. > I not only dispose the object within CallBack but I also dispose > the reference that was passed again just incase.. each BeginInvoke on a delegate? Failing to do so would result in memory leak... Arnaud MVP - VC
To start with, I think you are just too anxious. If you have read MSDN, there
is an article that explains that the .Net GC is based on a liberal algorithm (mark and remove). This algorithm works on a stupid belief that the memory in the system is infinite. Because memory management is automatic in the CLR, the GC needs some way to determine which objects can be collected and which not. To help GC determine disposable objects, it is instructed that all objects in an idle state or which are no longer referenced by any application can be safely considered as disposable objects. But, because the managed heap is made up of contiguous memory locations, those objects which have been removed from memory would leave a void in between causing performance leaks for applications that require faster object allocation, the GC is also told to compact these void areas so as to maintain contiguity for enabling faster access to free memory for allocating new objects to memory. Another important aspect of how the managed heap is managed by the GC is through the generations. Usually, the managed memory is divided into three generations - 0,1,2. Generation 0 is considered to be the critical area for the GC. It has been instructed always to look up Gen 0 and if memory is critical in this generation then it is instructed to move all old objects to the subsequent generation to make room for fresh objects. This movement of objects is called promotion of objects. But this promotion happens only if there is a collection in a particular generation. For example, if you execute a statement GC.Collect(0) then when GC is invoked a collect will be performed on Generation 0 causing older objects to be moved from gen 0 to 1. If you perform another collection with GC.Collect(1) then objects from Gen 1 will be moved to Gen 2. But, this is where you are seeking info, since there is no more generation, the Gen 2 will keep expanding until the system is out of memory and an OutOfMemoryException is thrown. Instead of Ants and the other tools that you are using, I suggest you use the CLR Profiler (search on clrprofiler.exe free download on Google) which provides you with a Heap Dump through which you can get many perspectives into how the CLR is managing memory within. with regards, J.V. [quoted text, click to view] "CodeGuru" wrote:
> Althought I have read hundreds if not more pages on > msdn as well other sites on IDisposable and heap > generations. I am still a bit confused as to why > the application I have written is acting in such > "weird/awkward" manner. > > Basic outline of the application goes: > > Multi threaded socket application using Asynchronous > writes/reads/connects via TCP Sockets. > > I have written an application that starts 100 threads > and attaches 10 sockets from a peer to each of these > threads; thus 1000 active sockets open. > > Now. the application runs fine; I calculate using ANTS profiler > or any other profiling utility and it tells me that I have approximately > 30 MB of Memory used. > > I call GC.GetTotalMemory(false) or even (true) and it gives me approximately > the same values depending what stage of processing each thread > is at. > > All threads are doing the same thing by the way. > > When I look in Windows Performance Monitor and under .NET Memory > for that particular application the memory seems to be quite high. > > Several Hundred MEGABYTES. > > The application runs over time.. slowly but surely increasing in memory. > as the threads complete their tasks with the sockets it closes them. > and Disposes each and every object. I explicitly set each object to null > as well its children.. and have inherited IDisposable on almost all > objects. > > What I don't understand is why the memory keeps increasing, > while the application keeps reporting only 30MB of usage... the memory > keeps increasing and increasing until the system runs out of Physical Memory > and causes OutOfMemoryExpections.... its quite strange. I am not doing > anything within the application that would create anything larger then it > should. > > However I do thousands of XmlSerializations using a single MemoryStream > for each socket that is created; there are thousands of Asychronous callbacks > that are happening. > > My Generation 0 and 1 heaps seem to be doing just fine.. but 2nd gen heap > seems to be at hundreds of megs; its very strange to see this. > > I assumed that the .NET GC will keep on expanding the space required until > it is ready for a garbage collection.. would it be because of the magnitute of > ArrayLists that I am using ? or perhaps there are scattered memory segments. > throughout the Heaps ? is the GC having a difficult time collecting all the > resources? > > Any help on why this is occuring would be appreciated. > > I have tried my best to clean up all the resources.. but I figure that > most of the objects created should automatically get cleaned up > after it is out of the final scope that it is passed to.. for example. > once a command comes in via the Async Read Callback, I process it > and pass it to the appropriate thread DataReceive method. > I not only dispose the object within CallBack but I also dispose > the reference that was passed again just incase.. > > doesn't seem to be doing anything .. and what is even more strange > is why is .NET or all Profilers such as ANTS .NET profiler report > that I am only using 30MB of memory ? when in Task Manager > or Performance Monitor I see hundreds of megs like 600+. >
Ravichandran,
Thanks for the info, I know of how generations work; but regardless still useful information. A question I have is when looking at performance monitor I noticed hundreds if not thousands of objects being pinned, my point being is why would objects get pinned in the first place ? is it because perhaps it is being access by the system ? perhaps there is a lock on the object ? would locked objects cause the GC to pin the object and not be able to compact the Heap ? I haven't been able to find proper documentation on the different reasons why objects would get pinned. All in all, it seems by looking at the CLR Profiler that there is many voids (empty spaces) in the memory is this normal ? I also noticed the TOTAL HEAP SIZE of my application remainds between 500 to 600 MB and never expands more then that.. which is a good thing. but ! there is always a but.. when looking at the WORKINGSET it seems to be expanding and expanding.. also when looking at Task Manager the MEM USAGE is always stable yet the VM SIZE keeps growing ? the total available memory seems to be keep growing until I minimize the window? once doing so, the memory drastically grows from 200 MB to 800 to 900 MB of free space ? Would Console.WriteLine calls perhaps have a buffer problem ? or perhaps using Reflections to reflect and create objects in thousands per second cause a memory problem ? The communication layer in my application uses reflections to grab the root xml node and create an object based on that name and deserialize it, I wonder if this could perhaps also be a cause of this problem ? I am still looking at the CLR its a bit hard to follow, but nevertheless useful. Thank you for the info Kaz [quoted text, click to view] "Ravichandran J.V." wrote:
> To start with, I think you are just too anxious. If you have read MSDN, there > is an article that explains that the .Net GC is based on a liberal algorithm > (mark and remove). This algorithm works on a stupid belief that the memory in > the system is infinite. > > Because memory management is automatic in the CLR, the GC needs some way to > determine which objects can be collected and which not. To help GC determine > disposable objects, it is instructed that all objects in an idle state or > which are no longer referenced by any application can be safely considered as > disposable objects. > > But, because the managed heap is made up of contiguous memory locations, > those objects which have been removed from memory would leave a void in > between causing performance leaks for applications that require faster object > allocation, the GC is also told to compact these void areas so as to maintain > contiguity for enabling faster access to free memory for allocating new > objects to memory. Another important aspect of how the managed heap is > managed by the GC is through the generations. Usually, the managed memory is > divided into three generations - 0,1,2. Generation 0 is considered to be the > critical area for the GC. It has been instructed always to look up Gen 0 and > if memory is critical in this generation then it is instructed to move all > old objects to the subsequent generation to make room for fresh objects. This > movement of objects is called promotion of objects. But this promotion > happens only if there is a collection in a particular generation. For > example, if you execute a statement GC.Collect(0) then when GC is invoked a > collect will be performed on Generation 0 causing older objects to be moved > from gen 0 to 1. If you perform another collection with GC.Collect(1) then > objects from Gen 1 will be moved to Gen 2. > > But, this is where you are seeking info, since there is no more generation, > the Gen 2 will keep expanding until the system is out of memory and an > OutOfMemoryException is thrown. > > Instead of Ants and the other tools that you are using, I suggest you use > the CLR Profiler (search on clrprofiler.exe free download on Google) which > provides you with a Heap Dump through which you can get many perspectives > into how the CLR is managing memory within. > > with regards, > > J.V. > > > > > "CodeGuru" wrote: > > > Althought I have read hundreds if not more pages on > > msdn as well other sites on IDisposable and heap > > generations. I am still a bit confused as to why > > the application I have written is acting in such > > "weird/awkward" manner. > > > > Basic outline of the application goes: > > > > Multi threaded socket application using Asynchronous > > writes/reads/connects via TCP Sockets. > > > > I have written an application that starts 100 threads > > and attaches 10 sockets from a peer to each of these > > threads; thus 1000 active sockets open. > > > > Now. the application runs fine; I calculate using ANTS profiler > > or any other profiling utility and it tells me that I have approximately > > 30 MB of Memory used. > > > > I call GC.GetTotalMemory(false) or even (true) and it gives me approximately > > the same values depending what stage of processing each thread > > is at. > > > > All threads are doing the same thing by the way. > > > > When I look in Windows Performance Monitor and under .NET Memory > > for that particular application the memory seems to be quite high. > > > > Several Hundred MEGABYTES. > > > > The application runs over time.. slowly but surely increasing in memory. > > as the threads complete their tasks with the sockets it closes them. > > and Disposes each and every object. I explicitly set each object to null > > as well its children.. and have inherited IDisposable on almost all > > objects. > > > > What I don't understand is why the memory keeps increasing, > > while the application keeps reporting only 30MB of usage... the memory > > keeps increasing and increasing until the system runs out of Physical Memory > > and causes OutOfMemoryExpections.... its quite strange. I am not doing > > anything within the application that would create anything larger then it > > should. > > > > However I do thousands of XmlSerializations using a single MemoryStream > > for each socket that is created; there are thousands of Asychronous callbacks > > that are happening. > > > > My Generation 0 and 1 heaps seem to be doing just fine.. but 2nd gen heap > > seems to be at hundreds of megs; its very strange to see this. > > > > I assumed that the .NET GC will keep on expanding the space required until > > it is ready for a garbage collection.. would it be because of the magnitute of > > ArrayLists that I am using ? or perhaps there are scattered memory segments. > > throughout the Heaps ? is the GC having a difficult time collecting all the > > resources? > > > > Any help on why this is occuring would be appreciated. > >
Willy,
Thanks for the info, now that you mention it, and reviewing your posting for a second time, I noticed something you mentioned !! that could possibly .. maybe be a serious problem ! you said "I am holding onto buffers in my read / write" when infact and indeed I am.. I have created a private member for each socket that has a Buffer for reading and a ObjectBuffer for completing. once a full object has been completed I copy it to a completed buffer and execute a Reflection on it to "create" the actual object from the serialization stream (socket) more over, the object buffer never really gets destroyed unless the socket is killed.. so perhaps your telling me that because I have these infact may be the pinned objects? in my application ? Also: pinned object in performance monitor shows drastic values; it goes anywhere between 0 to 2000 pinned objects on each update. Private Bytes in Process shows over 300 MB for Active 500 Sockets... and 50 threads. Kaz [quoted text, click to view] "Willy Denoyette [MVP]" wrote:
> > "CodeGuru" <CodeGuru@discussions.microsoft.com> wrote in message > news:602B1F08-9484-420C-B157-2652E54E046B@microsoft.com... > > Althought I have read hundreds if not more pages on > > msdn as well other sites on IDisposable and heap > > generations. I am still a bit confused as to why > > the application I have written is acting in such > > "weird/awkward" manner. > > > > Basic outline of the application goes: > > > > Multi threaded socket application using Asynchronous > > writes/reads/connects via TCP Sockets. > > > > I have written an application that starts 100 threads > > and attaches 10 sockets from a peer to each of these > > threads; thus 1000 active sockets open. > > > > Now. the application runs fine; I calculate using ANTS profiler > > or any other profiling utility and it tells me that I have approximately > > 30 MB of Memory used. > > > > I call GC.GetTotalMemory(false) or even (true) and it gives me > > approximately > > the same values depending what stage of processing each thread > > is at. > > > > All threads are doing the same thing by the way. > > > > When I look in Windows Performance Monitor and under .NET Memory > > for that particular application the memory seems to be quite high. > > > > Several Hundred MEGABYTES. > > > > The application runs over time.. slowly but surely increasing in memory. > > as the threads complete their tasks with the sockets it closes them. > > and Disposes each and every object. I explicitly set each object to null > > as well its children.. and have inherited IDisposable on almost all > > objects. > > > > What I don't understand is why the memory keeps increasing, > > while the application keeps reporting only 30MB of usage... the memory > > keeps increasing and increasing until the system runs out of Physical > > Memory > > and causes OutOfMemoryExpections.... its quite strange. I am not doing > > anything within the application that would create anything larger then it > > should. > > > > However I do thousands of XmlSerializations using a single MemoryStream > > for each socket that is created; there are thousands of Asychronous > > callbacks > > that are happening. > > > > My Generation 0 and 1 heaps seem to be doing just fine.. but 2nd gen heap > > seems to be at hundreds of megs; its very strange to see this. > > > > I assumed that the .NET GC will keep on expanding the space required until > > it is ready for a garbage collection.. would it be because of the > > magnitute of > > ArrayLists that I am using ? or perhaps there are scattered memory > > segments. > > throughout the Heaps ? is the GC having a difficult time collecting all > > the > > resources? > > > > Any help on why this is occuring would be appreciated. > > > > I have tried my best to clean up all the resources.. but I figure that > > most of the objects created should automatically get cleaned up > > after it is out of the final scope that it is passed to.. for example. > > once a command comes in via the Async Read Callback, I process it > > and pass it to the appropriate thread DataReceive method. > > I not only dispose the object within CallBack but I also dispose > > the reference that was passed again just incase.. > > > > doesn't seem to be doing anything .. and what is even more strange > > is why is .NET or all Profilers such as ANTS .NET profiler report > > that I am only using 30MB of memory ? when in Task Manager > > or Performance Monitor I see hundreds of megs like 600+. > > > > Thank you and great weekend to all. > > 1. Don't use taskman, use perfmon and the "CLR Memory" counters Gen0, 1 and > 2 size to watch managed memory consumption. > 2. Take a look at the # of pinned objects using perfmon (CLR memory > counters). > 3. Use perfmon to watch your "process" privates bytes. > > If the the Gen 2 counter indicates a steady growth and the number of pinned > objects is rather high (> 100) chances are that pinned objects are aging > into Gen2 and as such preventing compactation of the heap. > If that's the case you are probably holding references to socket > (read/write) buffers for the lifetime of the socket. > > I would also suggest you to reduce the number of threads, 100 threads is > only a waste of resources (both memory and CPU) when using asynchronous > sockets. > > Willy. > > > >
<quote>I haven't been able to find proper documentation on the different
reasons why objects would get pinned.</quote> Objects would get pinned while, as you have surmised, they have been locked; they would get pinned if the object is being used by an unmanaged app or object; they would get pinned if you use the fixed expression to pin unsafe code. [quoted text, click to view] > but ! there is always a but.. when looking at the WORKINGSET > it seems to be expanding and expanding.. also when looking at > Task Manager the MEM USAGE is always stable yet the VM SIZE > keeps growing ? Windows uses upto 4 GB of virtual memory according to my coleague who is a MCSE. with regards, J.v. [quoted text, click to view] "CodeGuru" wrote:
> Ravichandran, > > Thanks for the info, I know of how generations work; but regardless > still useful information. A question I have is when looking at performance > monitor I noticed hundreds if not thousands of objects being pinned, > my point being is why would objects get pinned in the first place ? > is it because perhaps it is being access by the system ? perhaps > there is a lock on the object ? would locked objects cause the GC > to pin the object and not be able to compact the Heap ? > > I haven't been able to find proper documentation on the different > reasons why objects would get pinned. > > All in all, it seems by looking at the CLR Profiler that there is many > voids (empty spaces) in the memory is this normal ? > > I also noticed the TOTAL HEAP SIZE of my application remainds between > 500 to 600 MB and never expands more then that.. which is a good thing. > but ! there is always a but.. when looking at the WORKINGSET > it seems to be expanding and expanding.. also when looking at Task Manager > the MEM USAGE is always stable yet the VM SIZE keeps growing ? > the total available memory seems to be keep growing until I minimize the > window? > once doing so, the memory drastically grows from 200 MB to 800 to 900 MB of > free space ? > > Would Console.WriteLine calls perhaps have a buffer problem ? > or perhaps using Reflections to reflect and create objects in thousands > per second cause a memory problem ? > > The communication layer in my application uses reflections to grab the root > xml node and create an object based on that name and deserialize it, I > wonder if this could perhaps also be a cause of this problem ? > > I am still looking at the CLR its a bit hard to follow, but nevertheless > useful. > > Thank you for the info > > > Kaz > > > "Ravichandran J.V." wrote: > > > To start with, I think you are just too anxious. If you have read MSDN, there > > is an article that explains that the .Net GC is based on a liberal algorithm > > (mark and remove). This algorithm works on a stupid belief that the memory in > > the system is infinite. > > > > Because memory management is automatic in the CLR, the GC needs some way to > > determine which objects can be collected and which not. To help GC determine > > disposable objects, it is instructed that all objects in an idle state or > > which are no longer referenced by any application can be safely considered as > > disposable objects. > > > > But, because the managed heap is made up of contiguous memory locations, > > those objects which have been removed from memory would leave a void in > > between causing performance leaks for applications that require faster object > > allocation, the GC is also told to compact these void areas so as to maintain > > contiguity for enabling faster access to free memory for allocating new > > objects to memory. Another important aspect of how the managed heap is > > managed by the GC is through the generations. Usually, the managed memory is > > divided into three generations - 0,1,2. Generation 0 is considered to be the > > critical area for the GC. It has been instructed always to look up Gen 0 and > > if memory is critical in this generation then it is instructed to move all > > old objects to the subsequent generation to make room for fresh objects. This > > movement of objects is called promotion of objects. But this promotion > > happens only if there is a collection in a particular generation. For > > example, if you execute a statement GC.Collect(0) then when GC is invoked a > > collect will be performed on Generation 0 causing older objects to be moved > > from gen 0 to 1. If you perform another collection with GC.Collect(1) then > > objects from Gen 1 will be moved to Gen 2. > > > > But, this is where you are seeking info, since there is no more generation, > > the Gen 2 will keep expanding until the system is out of memory and an > > OutOfMemoryException is thrown. > > > > Instead of Ants and the other tools that you are using, I suggest you use > > the CLR Profiler (search on clrprofiler.exe free download on Google) which > > provides you with a Heap Dump through which you can get many perspectives > > into how the CLR is managing memory within. > > > > with regards, > > > > J.V. > > > > > > > > > > "CodeGuru" wrote: > > > > > Althought I have read hundreds if not more pages on > > > msdn as well other sites on IDisposable and heap > > > generations. I am still a bit confused as to why > > > the application I have written is acting in such > > > "weird/awkward" manner. > > > > > > Basic outline of the application goes: > > > > > > Multi threaded socket application using Asynchronous > > > writes/reads/connects via TCP Sockets. > > > > > > I have written an application that starts 100 threads > > > and attaches 10 sockets from a peer to each of these > > > threads; thus 1000 active sockets open. > > > > > > Now. the application runs fine; I calculate using ANTS profiler > > > or any other profiling utility and it tells me that I have approximately > > > 30 MB of Memory used. > > > > > > I call GC.GetTotalMemory(false) or even (true) and it gives me approximately > > > the same values depending what stage of processing each thread > > > is at. > > > > > > All threads are doing the same thing by the way. > > > > > > When I look in Windows Performance Monitor and under .NET Memory > > > for that particular application the memory seems to be quite high. > > > > > > Several Hundred MEGABYTES. > > > > > > The application runs over time.. slowly but surely increasing in memory. > > > as the threads complete their tasks with the sockets it closes them. > > > and Disposes each and every object. I explicitly set each object to null > > > as well its children.. and have inherited IDisposable on almost all > > > objects. > > > > > > What I don't understand is why the memory keeps increasing, > > > while the application keeps reporting only 30MB of usage... the memory > > > keeps increasing and increasing until the system runs out of Physical Memory
A couple of sanity-check questions:
1. You're not running Debug builds are you? 2. Have you turned on the Server framework with COMPLUS_BUILDFLAVOR or something? -- Phil Wilson [MVP Windows Installer] ---- [quoted text, click to view] "CodeGuru" <CodeGuru@discussions.microsoft.com> wrote in message
news:BBB3F5A8-2358-4CC6-9A35-DF792E6EBF97@microsoft.com... > Ravichandran, > > Thanks for the info, I know of how generations work; but regardless > still useful information. A question I have is when looking at performance > monitor I noticed hundreds if not thousands of objects being pinned, > my point being is why would objects get pinned in the first place ? > is it because perhaps it is being access by the system ? perhaps > there is a lock on the object ? would locked objects cause the GC > to pin the object and not be able to compact the Heap ? > > I haven't been able to find proper documentation on the different > reasons why objects would get pinned. > > All in all, it seems by looking at the CLR Profiler that there is many > voids (empty spaces) in the memory is this normal ? > > I also noticed the TOTAL HEAP SIZE of my application remainds between > 500 to 600 MB and never expands more then that.. which is a good thing. > but ! there is always a but.. when looking at the WORKINGSET > it seems to be expanding and expanding.. also when looking at Task Manager > the MEM USAGE is always stable yet the VM SIZE keeps growing ? > the total available memory seems to be keep growing until I minimize the > window? > once doing so, the memory drastically grows from 200 MB to 800 to 900 MB > of > free space ? > > Would Console.WriteLine calls perhaps have a buffer problem ? > or perhaps using Reflections to reflect and create objects in thousands > per second cause a memory problem ? > > The communication layer in my application uses reflections to grab the > root > xml node and create an object based on that name and deserialize it, I > wonder if this could perhaps also be a cause of this problem ? > > I am still looking at the CLR its a bit hard to follow, but nevertheless > useful. > > Thank you for the info > > > Kaz > > > "Ravichandran J.V." wrote: > >> To start with, I think you are just too anxious. If you have read MSDN, >> there >> is an article that explains that the .Net GC is based on a liberal >> algorithm >> (mark and remove). This algorithm works on a stupid belief that the >> memory in >> the system is infinite. >> >> Because memory management is automatic in the CLR, the GC needs some way >> to >> determine which objects can be collected and which not. To help GC >> determine >> disposable objects, it is instructed that all objects in an idle state or >> which are no longer referenced by any application can be safely >> considered as >> disposable objects. >> >> But, because the managed heap is made up of contiguous memory locations, >> those objects which have been removed from memory would leave a void in >> between causing performance leaks for applications that require faster >> object >> allocation, the GC is also told to compact these void areas so as to >> maintain >> contiguity for enabling faster access to free memory for allocating new >> objects to memory. Another important aspect of how the managed heap is >> managed by the GC is through the generations. Usually, the managed memory >> is >> divided into three generations - 0,1,2. Generation 0 is considered to be >> the >> critical area for the GC. It has been instructed always to look up Gen 0 >> and >> if memory is critical in this generation then it is instructed to move >> all >> old objects to the subsequent generation to make room for fresh objects. >> This >> movement of objects is called promotion of objects. But this promotion >> happens only if there is a collection in a particular generation. For >> example, if you execute a statement GC.Collect(0) then when GC is invoked >> a >> collect will be performed on Generation 0 causing older objects to be >> moved >> from gen 0 to 1. If you perform another collection with GC.Collect(1) >> then >> objects from Gen 1 will be moved to Gen 2. >> >> But, this is where you are seeking info, since there is no more >> generation, >> the Gen 2 will keep expanding until the system is out of memory and an >> OutOfMemoryException is thrown. >> >> Instead of Ants and the other tools that you are using, I suggest you use >> the CLR Profiler (search on clrprofiler.exe free download on Google) >> which >> provides you with a Heap Dump through which you can get many perspectives >> into how the CLR is managing memory within. >> >> with regards, >> >> J.V. >> >> >> >> >> "CodeGuru" wrote: >> >> > Althought I have read hundreds if not more pages on >> > msdn as well other sites on IDisposable and heap >> > generations. I am still a bit confused as to why >> > the application I have written is acting in such >> > "weird/awkward" manner. >> > >> > Basic outline of the application goes: >> > >> > Multi threaded socket application using Asynchronous >> > writes/reads/connects via TCP Sockets. >> > >> > I have written an application that starts 100 threads >> > and attaches 10 sockets from a peer to each of these >> > threads; thus 1000 active sockets open. >> > >> > Now. the application runs fine; I calculate using ANTS profiler >> > or any other profiling utility and it tells me that I have >> > approximately >> > 30 MB of Memory used. >> > >> > I call GC.GetTotalMemory(false) or even (true) and it gives me >> > approximately >> > the same values depending what stage of processing each thread >> > is at. >> > >> > All threads are doing the same thing by the way. >> > >> > When I look in Windows Performance Monitor and under .NET Memory >> > for that particular application the memory seems to be quite high. >> > >> > Several Hundred MEGABYTES. >> > >> > The application runs over time.. slowly but surely increasing in >> > memory. >> > as the threads complete their tasks with the sockets it closes them. >> > and Disposes each and every object. I explicitly set each object to >> > null >> > as well its children.. and have inherited IDisposable on almost all >> > objects. >> > >> > What I don't understand is why the memory keeps increasing, >> > while the application keeps reporting only 30MB of usage... the memory >> > keeps increasing and increasing until the system runs out of Physical >> > Memory >> > and causes OutOfMemoryExpections.... its quite strange. I am not doing >> > anything within the application that would create anything larger then >> > it >> > should. >> > >> > However I do thousands of XmlSerializations using a single MemoryStream >> > for each socket that is created; there are thousands of Asychronous >> > callbacks >> > that are happening. >> >
Phil,
I have ran in both Debug and Release, same results and no I haven't turned on COMPLUS_BUILDFLAVOR, how would go about doing this (sorry for the ignorance) ? Using VS 2003. Thanks Kaz [quoted text, click to view] "Phil Wilson" wrote:
> A couple of sanity-check questions: > 1. You're not running Debug builds are you? > 2. Have you turned on the Server framework with COMPLUS_BUILDFLAVOR or > something? > -- > Phil Wilson [MVP Windows Installer] > ---- > "CodeGuru" <CodeGuru@discussions.microsoft.com> wrote in message > news:BBB3F5A8-2358-4CC6-9A35-DF792E6EBF97@microsoft.com... > > Ravichandran, > > > > Thanks for the info, I know of how generations work; but regardless > > still useful information. A question I have is when looking at performance > > monitor I noticed hundreds if not thousands of objects being pinned, > > my point being is why would objects get pinned in the first place ? > > is it because perhaps it is being access by the system ? perhaps > > there is a lock on the object ? would locked objects cause the GC > > to pin the object and not be able to compact the Heap ? > > > > I haven't been able to find proper documentation on the different > > reasons why objects would get pinned. > > > > All in all, it seems by looking at the CLR Profiler that there is many > > voids (empty spaces) in the memory is this normal ? > > > > I also noticed the TOTAL HEAP SIZE of my application remainds between > > 500 to 600 MB and never expands more then that.. which is a good thing. > > but ! there is always a but.. when looking at the WORKINGSET > > it seems to be expanding and expanding.. also when looking at Task Manager > > the MEM USAGE is always stable yet the VM SIZE keeps growing ? > > the total available memory seems to be keep growing until I minimize the > > window? > > once doing so, the memory drastically grows from 200 MB to 800 to 900 MB > > of > > free space ? > > > > Would Console.WriteLine calls perhaps have a buffer problem ? > > or perhaps using Reflections to reflect and create objects in thousands > > per second cause a memory problem ? > > > > The communication layer in my application uses reflections to grab the > > root > > xml node and create an object based on that name and deserialize it, I > > wonder if this could perhaps also be a cause of this problem ? > > > > I am still looking at the CLR its a bit hard to follow, but nevertheless > > useful. > > > > Thank you for the info > > > > > > Kaz > > > > > > "Ravichandran J.V." wrote: > > > >> To start with, I think you are just too anxious. If you have read MSDN, > >> there > >> is an article that explains that the .Net GC is based on a liberal > >> algorithm > >> (mark and remove). This algorithm works on a stupid belief that the > >> memory in > >> the system is infinite. > >> > >> Because memory management is automatic in the CLR, the GC needs some way > >> to > >> determine which objects can be collected and which not. To help GC > >> determine > >> disposable objects, it is instructed that all objects in an idle state or > >> which are no longer referenced by any application can be safely > >> considered as > >> disposable objects. > >> > >> But, because the managed heap is made up of contiguous memory locations, > >> those objects which have been removed from memory would leave a void in > >> between causing performance leaks for applications that require faster > >> object > >> allocation, the GC is also told to compact these void areas so as to > >> maintain > >> contiguity for enabling faster access to free memory for allocating new > >> objects to memory. Another important aspect of how the managed heap is > >> managed by the GC is through the generations. Usually, the managed memory > >> is > >> divided into three generations - 0,1,2. Generation 0 is considered to be > >> the > >> critical area for the GC. It has been instructed always to look up Gen 0 > >> and > >> if memory is critical in this generation then it is instructed to move > >> all > >> old objects to the subsequent generation to make room for fresh objects. > >> This > >> movement of objects is called promotion of objects. But this promotion > >> happens only if there is a collection in a particular generation. For > >> example, if you execute a statement GC.Collect(0) then when GC is invoked > >> a > >> collect will be performed on Generation 0 causing older objects to be > >> moved > >> from gen 0 to 1. If you perform another collection with GC.Collect(1) > >> then > >> objects from Gen 1 will be moved to Gen 2. > >> > >> But, this is where you are seeking info, since there is no more > >> generation, > >> the Gen 2 will keep expanding until the system is out of memory and an > >> OutOfMemoryException is thrown. > >> > >> Instead of Ants and the other tools that you are using, I suggest you use > >> the CLR Profiler (search on clrprofiler.exe free download on Google) > >> which > >> provides you with a Heap Dump through which you can get many perspectives > >> into how the CLR is managing memory within. > >> > >> with regards, > >> > >> J.V. > >> > >> > >> > >> > >> "CodeGuru" wrote: > >> > >> > Althought I have read hundreds if not more pages on > >> > msdn as well other sites on IDisposable and heap > >> > generations. I am still a bit confused as to why > >> > the application I have written is acting in such > >> > "weird/awkward" manner. > >> > > >> > Basic outline of the application goes: > >> > > >> > Multi threaded socket application using Asynchronous > >> > writes/reads/connects via TCP Sockets. > >> > > >> > I have written an application that starts 100 threads > >> > and attaches 10 sockets from a peer to each of these > >> > threads; thus 1000 active sockets open. > >> > > >> > Now. the application runs fine; I calculate using ANTS profiler > >> > or any other profiling utility and it tells me that I have > >> > approximately > >> > 30 MB of Memory used. > >> > > >> > I call GC.GetTotalMemory(false) or even (true) and it gives me > >> > approximately > >> > the same values depending what stage of processing each thread > >> > is at. > >> > > >> > All threads are doing the same thing by the way. > >> > > >> > When I look in Windows Performance Monitor and under .NET Memory > >> > for that particular application the memory seems to be quite high. > >> > > >> > Several Hundred MEGABYTES. > >> > > >> > The application runs over time.. slowly but surely increasing in > >> > memory. > >> > as the threads complete their tasks with the sockets it closes them. > >> > and Disposes each and every object. I explicitly set each object to > >> > null > >> > as well its children.. and have inherited IDisposable on almost all > >> > objects. > >> > > >> > What I don't understand is why the memory keeps increasing,
<quote> and create objects in thousands per second cause a memory problem
</quote> Memory problems do not arise in the managed environment due to numerous objects but due to numerous GC collections. Get a view of the number of GC collections on a generation in the CLR Profiler. More the collections on a generation, more you should think about how objects are being used by your app. with regards, J.V. [quoted text, click to view] "Ravichandran J.V." wrote:
> <quote>I haven't been able to find proper documentation on the different > reasons why objects would get pinned.</quote> > > Objects would get pinned while, as you have surmised, they have been locked; > they would get pinned if the object is being used by an unmanaged app or > object; they would get pinned if you use the fixed expression to pin unsafe > code. > > > but ! there is always a but.. when looking at the WORKINGSET > > it seems to be expanding and expanding.. also when looking at > > Task Manager the MEM USAGE is always stable yet the VM SIZE > > keeps growing ? > > Windows uses upto 4 GB of virtual memory according to my coleague who is a > MCSE. > > with regards, > > J.v. > > > "CodeGuru" wrote: > > > Ravichandran, > > > > Thanks for the info, I know of how generations work; but regardless > > still useful information. A question I have is when looking at performance > > monitor I noticed hundreds if not thousands of objects being pinned, > > my point being is why would objects get pinned in the first place ? > > is it because perhaps it is being access by the system ? perhaps > > there is a lock on the object ? would locked objects cause the GC > > to pin the object and not be able to compact the Heap ? > > > > I haven't been able to find proper documentation on the different > > reasons why objects would get pinned. > > > > All in all, it seems by looking at the CLR Profiler that there is many > > voids (empty spaces) in the memory is this normal ? > > > > I also noticed the TOTAL HEAP SIZE of my application remainds between > > 500 to 600 MB and never expands more then that.. which is a good thing. > > but ! there is always a but.. when looking at the WORKINGSET > > it seems to be expanding and expanding.. also when looking at Task Manager > > the MEM USAGE is always stable yet the VM SIZE keeps growing ? > > the total available memory seems to be keep growing until I minimize the > > window? > > once doing so, the memory drastically grows from 200 MB to 800 to 900 MB of > > free space ? > > > > Would Console.WriteLine calls perhaps have a buffer problem ? > > or perhaps using Reflections to reflect and create objects in thousands > > per second cause a memory problem ? > > > > The communication layer in my application uses reflections to grab the root > > xml node and create an object based on that name and deserialize it, I > > wonder if this could perhaps also be a cause of this problem ? > > > > I am still looking at the CLR its a bit hard to follow, but nevertheless > > useful. > > > > Thank you for the info > > > > > > Kaz > > > > > > "Ravichandran J.V." wrote: > > > > > To start with, I think you are just too anxious. If you have read MSDN, there > > > is an article that explains that the .Net GC is based on a liberal algorithm > > > (mark and remove). This algorithm works on a stupid belief that the memory in > > > the system is infinite. > > > > > > Because memory management is automatic in the CLR, the GC needs some way to > > > determine which objects can be collected and which not. To help GC determine > > > disposable objects, it is instructed that all objects in an idle state or > > > which are no longer referenced by any application can be safely considered as > > > disposable objects. > > > > > > But, because the managed heap is made up of contiguous memory locations, > > > those objects which have been removed from memory would leave a void in > > > between causing performance leaks for applications that require faster object > > > allocation, the GC is also told to compact these void areas so as to maintain > > > contiguity for enabling faster access to free memory for allocating new > > > objects to memory. Another important aspect of how the managed heap is > > > managed by the GC is through the generations. Usually, the managed memory is > > > divided into three generations - 0,1,2. Generation 0 is considered to be the > > > critical area for the GC. It has been instructed always to look up Gen 0 and > > > if memory is critical in this generation then it is instructed to move all > > > old objects to the subsequent generation to make room for fresh objects. This > > > movement of objects is called promotion of objects. But this promotion > > > happens only if there is a collection in a particular generation. For > > > example, if you execute a statement GC.Collect(0) then when GC is invoked a > > > collect will be performed on Generation 0 causing older objects to be moved > > > from gen 0 to 1. If you perform another collection with GC.Collect(1) then > > > objects from Gen 1 will be moved to Gen 2. > > > > > > But, this is where you are seeking info, since there is no more generation, > > > the Gen 2 will keep expanding until the system is out of memory and an > > > OutOfMemoryException is thrown. > > > > > > Instead of Ants and the other tools that you are using, I suggest you use > > > the CLR Profiler (search on clrprofiler.exe free download on Google) which > > > provides you with a Heap Dump through which you can get many perspectives > > > into how the CLR is managing memory within. > > > > > > with regards, > > > > > > J.V. > > > > > > > > > > > > > > > "CodeGuru" wrote: > > > > > > > Althought I have read hundreds if not more pages on > > > > msdn as well other sites on IDisposable and heap > > > > generations. I am still a bit confused as to why > > > > the application I have written is acting in such > > > > "weird/awkward" manner. > > > > > > > > Basic outline of the application goes: > > > > > > > > Multi threaded socket application using Asynchronous > > > > writes/reads/connects via TCP Sockets. > > > > > > > > I have written an application that starts 100 threads > > > > and attaches 10 sockets from a peer to each of these > > > > threads; thus 1000 active sockets open. > > > > > > > > Now. the application runs fine; I calculate using ANTS profiler > > > > or any other profiling utility and it tells me that I have approximately > > > > 30 MB of Memory used. > > > > > > > > I call GC.GetTotalMemory(false) or even (true) and it gives me approximately > > > > the same values depending what stage of processing each thread > > > > is at. > > > > > > > > All threads are doing the same thing by the way. > > > >
Don't see what you're looking for? Try a search.
|
June 2003
July 2003
August 2003
September 2003
October 2003
November 2003
December 2003
January 2004
February 2004
March 2004
April 2004
May 2004
June 2004
July 2004
August 2004
September 2004
October 2004
November 2004
December 2004
January 2005
February 2005
March 2005
April 2005
May 2005
June 2005
July 2005
August 2005
September 2005
October 2005
November 2005
December 2005
January 2006
February 2006
March 2006
April 2006
May 2006
June 2006
July 2006
August 2006
September 2006
October 2006
November 2006
December 2006
January 2007
February 2007
March 2007
April 2007
May 2007
June 2007
July 2007
August 2007
September 2007
October 2007
November 2007
December 2007
January 2008
February 2008
March 2008
April 2008
May 2008
June 2008
| home · blog · groups | Questions? Comments? Contact the dn |