| Groups | Blog | Home |
dotnet remoting : Remoting callback hangs on ThreadPool thread
clients send messages to the server and the server sends messages via remote delegate callbacks to all clients. Remote delegate callbacks are invoked via BeginInvoke (ie. on a managed ThreadPool thread). The problem is that when the client is suspended, but not terminated (in a console app hit "pause" or suspend the process via SysInternals ProcessExplorer), the callback for .BeginInvoke is never fired and threadpool thread is never returned to the pool. Since there is no way to know that the client is "dead", the next server broadcast invokes the same remote delegate - 2nd threadpool thread is in limbo (3rd, 4th, etc). So when I suspend one client, I quicky run out of threadpool threads on the server and remoting dies. If I don't use BeginInvoke on remote delegate and instead create my own thread, the server will eventually create 1000s of threads and very bad things happen. Steps to reproduce: take any remoting client-server example with remote delegate callback; suspend the client; trigger the callback in a loop and watch your threads. Can anyone enlighten me on this issue of dealing with suspended remoting clients and remote server callbacks that never release a thread? I'd like to know why this happens and, more importantly, what the workaround is. P.S. [OneWay] attribute is not acceptable, since I need to know when the clients die. GenuineChannels is not an answer I'm looking for either. Thank you. -- Stanislav Drapkin sdrapkin at sdprime dot com
Stanislav,
read inline: In article <U5OdncFus6wnK77fRVn-rg@rogers.com>, "news.giganews.com" <sdrapkin at sdprime dot com> says... [quoted text, click to view] > I'm using remoting with TCP channel in a chat server scenario, ie. multiple > clients send messages to the server and the server sends messages via remote > delegate callbacks to all clients. Remote delegate callbacks are invoked via > BeginInvoke (ie. on a managed ThreadPool thread). > > The problem is that when the client is suspended, but not terminated (in a > console app hit "pause" or suspend the process via SysInternals > ProcessExplorer), the callback for .BeginInvoke is never fired and > threadpool thread is never returned to the pool. Since there is no way to This is not right, it is not never, but after a really long timeout, which you can not control. [quoted text, click to view] > know that the client is "dead", the next server broadcast invokes the same > remote delegate - 2nd threadpool thread is in limbo (3rd, 4th, etc). So when > I suspend one client, I quicky run out of threadpool threads on the server > and remoting dies. > > If I don't use BeginInvoke on remote delegate and instead create my own > thread, the server will eventually create 1000s of threads and very bad > things happen. > > Steps to reproduce: take any remoting client-server example with remote > delegate callback; suspend the client; trigger the callback in a loop and > watch your threads. > > Can anyone enlighten me on this issue of dealing with suspended remoting > clients and remote server callbacks that never release a thread? I'd like to > know why this happens and, more importantly, what the workaround is. > > P.S. [OneWay] attribute is not acceptable, since I need to know when the > clients die. GenuineChannels is not an answer I'm looking for either. The use the WaitOne approach. IAsyncResult ar = dlgt.BeginInvoke(...); if (ar.AsuncWaitHandle.WaitOne(timeout, false)) { //invocation finished //execute EndInvoke } else { //timeout expired //delete the client } Sunny [quoted text, click to view] >
> Thank you. > > -- > Stanislav Drapkin > sdrapkin at sdprime dot com > >
I have tried the WaitOne approach - it will only work if I serialize all
pending callbacks. Ie. one callback is triggered at a time, and when WaitOne returns false, the client is diconnected and other pending callbacks are cancelled. That defeats the asynchronous callback model. I guess I can live with one threadpool thread per client being lost for a "really long time which I cannot control", but is there a way to solve this without serializing the callbacks? -- Stanislav Drapkin sdrapkin at sdprime dot com [quoted text, click to view] "Sunny" <sunny@newsgroup.nospam> wrote in message news:eV3ENjoHFHA.3936@TK2MSFTNGP10.phx.gbl... > Stanislav, > read inline: > > In article <U5OdncFus6wnK77fRVn-rg@rogers.com>, "news.giganews.com" > <sdrapkin at sdprime dot com> says... >> I'm using remoting with TCP channel in a chat server scenario, ie. >> multiple >> clients send messages to the server and the server sends messages via >> remote >> delegate callbacks to all clients. Remote delegate callbacks are invoked >> via >> BeginInvoke (ie. on a managed ThreadPool thread). >> >> The problem is that when the client is suspended, but not terminated (in >> a >> console app hit "pause" or suspend the process via SysInternals >> ProcessExplorer), the callback for .BeginInvoke is never fired and >> threadpool thread is never returned to the pool. Since there is no way to > > This is not right, it is not never, but after a really long timeout, > which you can not control. > >> know that the client is "dead", the next server broadcast invokes the >> same >> remote delegate - 2nd threadpool thread is in limbo (3rd, 4th, etc). So >> when >> I suspend one client, I quicky run out of threadpool threads on the >> server >> and remoting dies. >> >> If I don't use BeginInvoke on remote delegate and instead create my own >> thread, the server will eventually create 1000s of threads and very bad >> things happen. >> >> Steps to reproduce: take any remoting client-server example with remote >> delegate callback; suspend the client; trigger the callback in a loop and >> watch your threads. >> >> Can anyone enlighten me on this issue of dealing with suspended remoting >> clients and remote server callbacks that never release a thread? I'd like >> to >> know why this happens and, more importantly, what the workaround is. >> >> P.S. [OneWay] attribute is not acceptable, since I need to know when the >> clients die. GenuineChannels is not an answer I'm looking for either. > > The use the WaitOne approach. > > IAsyncResult ar = dlgt.BeginInvoke(...); > > if (ar.AsuncWaitHandle.WaitOne(timeout, false)) > { > //invocation finished > //execute EndInvoke > } > else > { > //timeout expired > //delete the client > } > > Sunny > >> >> Thank you. >> >> -- >> Stanislav Drapkin >> sdrapkin at sdprime dot com >> >> >>
Unfortunately, they have forgot to implement a timeout value for TCP
channels. There are some solutions: 1. WaitOne (even if you don't like it) 2. using HTTP channels (of course with binary formatter), there you can set the connection timeout. 3. I haven't tried this, but sounds like a plan - every client creates a CAO on the server, and register this CAO as event receiver (or callback receiver). And this CAO will forward the callback to the client. Set this CAO's lifetime to something short, and register it to a client side sponsor. Then use TrackingServices to detect when the lifetime of the CAO is not renewed (the client is dead), and unregister it from the event chain. As far as I know, in v. 2.0 this problem with TCP channels is solved. Sunny In article <ivednV3XUbXUKLnfRVn-3w@rogers.com>, "news.giganews.com" <sdrapkin at sdprime dot com> says... [quoted text, click to view] > I have tried the WaitOne approach - it will only work if I serialize all
> pending callbacks. Ie. one callback is triggered at a time, and when WaitOne > returns false, the client is diconnected and other pending callbacks are > cancelled. > > That defeats the asynchronous callback model. > > I guess I can live with one threadpool thread per client being lost for a > "really long time which I cannot control", but is there a way to solve this > without serializing the callbacks? > > -- > Stanislav Drapkin > sdrapkin at sdprime dot com > > "Sunny" <sunny@newsgroup.nospam> wrote in message > news:eV3ENjoHFHA.3936@TK2MSFTNGP10.phx.gbl... > > Stanislav, > > read inline: > > > > In article <U5OdncFus6wnK77fRVn-rg@rogers.com>, "news.giganews.com" > > <sdrapkin at sdprime dot com> says... > >> I'm using remoting with TCP channel in a chat server scenario, ie. > >> multiple > >> clients send messages to the server and the server sends messages via > >> remote > >> delegate callbacks to all clients. Remote delegate callbacks are invoked > >> via > >> BeginInvoke (ie. on a managed ThreadPool thread). > >> > >> The problem is that when the client is suspended, but not terminated (in > >> a > >> console app hit "pause" or suspend the process via SysInternals > >> ProcessExplorer), the callback for .BeginInvoke is never fired and > >> threadpool thread is never returned to the pool. Since there is no way to > > > > This is not right, it is not never, but after a really long timeout, > > which you can not control. > > > >> know that the client is "dead", the next server broadcast invokes the > >> same > >> remote delegate - 2nd threadpool thread is in limbo (3rd, 4th, etc). So > >> when > >> I suspend one client, I quicky run out of threadpool threads on the > >> server > >> and remoting dies. > >> > >> If I don't use BeginInvoke on remote delegate and instead create my own > >> thread, the server will eventually create 1000s of threads and very bad > >> things happen. > >> > >> Steps to reproduce: take any remoting client-server example with remote > >> delegate callback; suspend the client; trigger the callback in a loop and > >> watch your threads. > >> > >> Can anyone enlighten me on this issue of dealing with suspended remoting > >> clients and remote server callbacks that never release a thread? I'd like > >> to > >> know why this happens and, more importantly, what the workaround is. > >> > >> P.S. [OneWay] attribute is not acceptable, since I need to know when the > >> clients die. GenuineChannels is not an answer I'm looking for either. > > > > The use the WaitOne approach. > > > > IAsyncResult ar = dlgt.BeginInvoke(...); > > > > if (ar.AsuncWaitHandle.WaitOne(timeout, false)) > > { > > //invocation finished > > //execute EndInvoke > > } > > else > > { > > //timeout expired > > //delete the client > > } > > > > Sunny > > > >> > >> Thank you. > >> > >> -- > >> Stanislav Drapkin > >> sdrapkin at sdprime dot com > >> > >> > >> > >
Thank you for your suggestions. Using HTTP channel is the only workaround I
know. Solution #3 is equivalent to #1 in that it helps you to mark the client as "disconnected" at some point in time, despite the missing (very very long) TCP channel timeout. However, when you get to that point in time, you may already have 25 (any number of) hanging threadpool threads -- unless you serialize the callbacks (back to square one). MSDN only mentions "the cause" - that TCP channel doesn't support configurable timeouts. I'm basically looking for a credible answer admitting the "consequence" - that asynchronous callbacks via TCP channel in .NET 1.1 do not work with suspended clients (ie. in any production environment). -- Stanislav Drapkin sdrapkin at sdprime dot com [quoted text, click to view] "Sunny" <sunny@newsgroup.nospam> wrote in message news:eIfZbhpHFHA.3332@TK2MSFTNGP14.phx.gbl... > Unfortunately, they have forgot to implement a timeout value for TCP > channels. There are some solutions: > 1. WaitOne (even if you don't like it) > 2. using HTTP channels (of course with binary formatter), there you can > set the connection timeout. > 3. I haven't tried this, but sounds like a plan - every client creates a > CAO on the server, and register this CAO as event receiver (or callback > receiver). And this CAO will forward the callback to the client. Set > this CAO's lifetime to something short, and register it to a client side > sponsor. Then use TrackingServices to detect when the lifetime of the > CAO is not renewed (the client is dead), and unregister it from the > event chain. > > As far as I know, in v. 2.0 this problem with TCP channels is solved. > > Sunny > > In article <ivednV3XUbXUKLnfRVn-3w@rogers.com>, "news.giganews.com" > <sdrapkin at sdprime dot com> says... >> I have tried the WaitOne approach - it will only work if I serialize all >> pending callbacks. Ie. one callback is triggered at a time, and when >> WaitOne >> returns false, the client is diconnected and other pending callbacks are >> cancelled. >> >> That defeats the asynchronous callback model. >> >> I guess I can live with one threadpool thread per client being lost for a >> "really long time which I cannot control", but is there a way to solve >> this >> without serializing the callbacks? >> >> -- >> Stanislav Drapkin >> sdrapkin at sdprime dot com >> >> "Sunny" <sunny@newsgroup.nospam> wrote in message >> news:eV3ENjoHFHA.3936@TK2MSFTNGP10.phx.gbl... >> > Stanislav, >> > read inline: >> > >> > In article <U5OdncFus6wnK77fRVn-rg@rogers.com>, "news.giganews.com" >> > <sdrapkin at sdprime dot com> says... >> >> I'm using remoting with TCP channel in a chat server scenario, ie. >> >> multiple >> >> clients send messages to the server and the server sends messages via >> >> remote >> >> delegate callbacks to all clients. Remote delegate callbacks are >> >> invoked >> >> via >> >> BeginInvoke (ie. on a managed ThreadPool thread). >> >> >> >> The problem is that when the client is suspended, but not terminated >> >> (in >> >> a >> >> console app hit "pause" or suspend the process via SysInternals >> >> ProcessExplorer), the callback for .BeginInvoke is never fired and >> >> threadpool thread is never returned to the pool. Since there is no way >> >> to >> > >> > This is not right, it is not never, but after a really long timeout, >> > which you can not control. >> > >> >> know that the client is "dead", the next server broadcast invokes the >> >> same >> >> remote delegate - 2nd threadpool thread is in limbo (3rd, 4th, etc). >> >> So >> >> when >> >> I suspend one client, I quicky run out of threadpool threads on the >> >> server >> >> and remoting dies. >> >> >> >> If I don't use BeginInvoke on remote delegate and instead create my >> >> own >> >> thread, the server will eventually create 1000s of threads and very >> >> bad >> >> things happen. >> >> >> >> Steps to reproduce: take any remoting client-server example with >> >> remote >> >> delegate callback; suspend the client; trigger the callback in a loop >> >> and >> >> watch your threads. >> >> >> >> Can anyone enlighten me on this issue of dealing with suspended >> >> remoting >> >> clients and remote server callbacks that never release a thread? I'd >> >> like >> >> to >> >> know why this happens and, more importantly, what the workaround is. >> >> >> >> P.S. [OneWay] attribute is not acceptable, since I need to know when >> >> the >> >> clients die. GenuineChannels is not an answer I'm looking for either. >> > >> > The use the WaitOne approach. >> > >> > IAsyncResult ar = dlgt.BeginInvoke(...); >> > >> > if (ar.AsuncWaitHandle.WaitOne(timeout, false)) >> > { >> > //invocation finished >> > //execute EndInvoke >> > } >> > else >> > { >> > //timeout expired >> > //delete the client >> > } >> > >> > Sunny >> > >> >> >> >> Thank you. >> >> >> >> -- >> >> Stanislav Drapkin >> >> sdrapkin at sdprime dot com >> >> >> >> >> >> >> >> >>
An alternative is to not use Remoting at all for situations like this.
Instead use messaging with a publish subscribe model. MSMQ or some other type of messaging service will allow you to simply send messages and the sender does not care if any one is listening. The sending is done asynchronously. See this article on pros and cons of remoting: http://www.thinktecture.com/Resources/RemotingFAQ/RemotingUseCases.html Event callbacks via remoting in general is not recommended by most people I talk to due to it's unreliability and lack of services supporting it. Carey "Stanislav Drapkin" <sdrapkin at sdprime dot com> wrote in news:RpmdnTU4CZbTQLnfRVn-uA@rogers.com: [quoted text, click to view] > Thank you for your suggestions. Using HTTP channel is the only
> workaround I know. > > Solution #3 is equivalent to #1 in that it helps you to mark the > client as "disconnected" at some point in time, despite the missing > (very very long) TCP channel timeout. However, when you get to that > point in time, you may already have 25 (any number of) hanging > threadpool threads -- unless you serialize the callbacks (back to > square one). > > MSDN only mentions "the cause" - that TCP channel doesn't support > configurable timeouts. I'm basically looking for a credible answer > admitting the "consequence" - that asynchronous callbacks via TCP > channel in .NET 1.1 do not work with suspended clients (ie. in any > production environment). -- > Stanislav Drapkin > sdrapkin at sdprime dot com > > "Sunny" <sunny@newsgroup.nospam> wrote in message > news:eIfZbhpHFHA.3332@TK2MSFTNGP14.phx.gbl... >> Unfortunately, they have forgot to implement a timeout value for TCP >> channels. There are some solutions: >> 1. WaitOne (even if you don't like it) >> 2. using HTTP channels (of course with binary formatter), there you >> can set the connection timeout. >> 3. I haven't tried this, but sounds like a plan - every client >> creates a CAO on the server, and register this CAO as event receiver >> (or callback receiver). And this CAO will forward the callback to the >> client. Set this CAO's lifetime to something short, and register it >> to a client side sponsor. Then use TrackingServices to detect when >> the lifetime of the CAO is not renewed (the client is dead), and >> unregister it from the event chain. >> >> As far as I know, in v. 2.0 this problem with TCP channels is solved. >> >> Sunny >> >> In article <ivednV3XUbXUKLnfRVn-3w@rogers.com>, "news.giganews.com" >> <sdrapkin at sdprime dot com> says... >>> I have tried the WaitOne approach - it will only work if I serialize >>> all pending callbacks. Ie. one callback is triggered at a time, and >>> when WaitOne >>> returns false, the client is diconnected and other pending callbacks >>> are cancelled. >>> >>> That defeats the asynchronous callback model. >>> >>> I guess I can live with one threadpool thread per client being lost >>> for a "really long time which I cannot control", but is there a way >>> to solve this >>> without serializing the callbacks? >>> >>> -- >>> Stanislav Drapkin >>> sdrapkin at sdprime dot com >>> >>> "Sunny" <sunny@newsgroup.nospam> wrote in message >>> news:eV3ENjoHFHA.3936@TK2MSFTNGP10.phx.gbl... >>> > Stanislav, >>> > read inline: >>> > >>> > In article <U5OdncFus6wnK77fRVn-rg@rogers.com>, >>> > "news.giganews.com" <sdrapkin at sdprime dot com> says... >>> >> I'm using remoting with TCP channel in a chat server scenario, >>> >> ie. multiple >>> >> clients send messages to the server and the server sends messages >>> >> via remote >>> >> delegate callbacks to all clients. Remote delegate callbacks are >>> >> invoked >>> >> via >>> >> BeginInvoke (ie. on a managed ThreadPool thread). >>> >> >>> >> The problem is that when the client is suspended, but not >>> >> terminated (in >>> >> a >>> >> console app hit "pause" or suspend the process via SysInternals >>> >> ProcessExplorer), the callback for .BeginInvoke is never fired >>> >> and threadpool thread is never returned to the pool. Since there >>> >> is no way to >>> > >>> > This is not right, it is not never, but after a really long >>> > timeout, which you can not control. >>> > >>> >> know that the client is "dead", the next server broadcast invokes >>> >> the same >>> >> remote delegate - 2nd threadpool thread is in limbo (3rd, 4th, >>> >> etc). So >>> >> when >>> >> I suspend one client, I quicky run out of threadpool threads on >>> >> the server >>> >> and remoting dies. >>> >> >>> >> If I don't use BeginInvoke on remote delegate and instead create >>> >> my own >>> >> thread, the server will eventually create 1000s of threads and >>> >> very bad >>> >> things happen. >>> >> >>> >> Steps to reproduce: take any remoting client-server example with >>> >> remote >>> >> delegate callback; suspend the client; trigger the callback in a >>> >> loop and >>> >> watch your threads. >>> >> >>> >> Can anyone enlighten me on this issue of dealing with suspended >>> >> remoting >>> >> clients and remote server callbacks that never release a thread? >>> >> I'd like >>> >> to >>> >> know why this happens and, more importantly, what the workaround >>> >> is. >>> >> >>> >> P.S. [OneWay] attribute is not acceptable, since I need to know >>> >> when the >>> >> clients die. GenuineChannels is not an answer I'm looking for >>> >> either. >>> > >>> > The use the WaitOne approach. >>> > >>> > IAsyncResult ar = dlgt.BeginInvoke(...); >>> > >>> > if (ar.AsuncWaitHandle.WaitOne(timeout, false)) >>> > { >>> > //invocation finished >>> > //execute EndInvoke >>> > } >>> > else >>> > { >>> > //timeout expired >>> > //delete the client >>> > } >>> > >>> > Sunny >>> > >>> >> >>> >> Thank you. >>> >> >>> >> -- >>> >> Stanislav Drapkin >>> >> sdrapkin at sdprime dot com >>> >> >>> >> >>> >> >>> >>> >>> > > >
Don't see what you're looking for? Try a search.
|
June 2003
July 2003
August 2003
September 2003
October 2003
November 2003
December 2003
January 2004
February 2004
March 2004
April 2004
May 2004
June 2004
July 2004
August 2004
September 2004
October 2004
November 2004
December 2004
January 2005
February 2005
March 2005
April 2005
May 2005
June 2005
July 2005
August 2005
September 2005
October 2005
November 2005
December 2005
January 2006
February 2006
March 2006
April 2006
May 2006
June 2006
July 2006
August 2006
September 2006
October 2006
November 2006
December 2006
January 2007
February 2007
March 2007
April 2007
May 2007
June 2007
July 2007
August 2007
September 2007
October 2007
November 2007
December 2007
January 2008
February 2008
March 2008
April 2008
May 2008
June 2008
| home · blog · groups | Questions? Comments? Contact the dn |