|
|
You're in the
dotnet web services enhancements
group:MTOM and WSE3
dotnet web services enhancements:
I am trying to return a 500Mb file from IIS to a winform client via a web service. I would like to use MTOM and have installed WSE3 on the client and server. I can see there is a chunking setting in the WSE3 settings set to 64K but I am not sure how I should get the chunking to work correctly. I have used some example code for the server setup as shown below but this seems to involve reading the full 500M into a byte array before returning it so WSE can take care of the chunking for me. I can see all the server memory being used up when I run the code and the client times out. Can you provide some example code and advise to efficiently transfer the file from the server to the client using chunking without using too much memory? <WebMethod(Description:="This is the MTOM test")> _ Public Function Test16() As Byte() Dim response As Byte() Dim filepath As String = AppDomain.CurrentDomain.BaseDirectory + "download\" + filename response = File.ReadAllBytes(filepath) Return response End Function
hi smarty
i wrote an article on codeproject which may be of use to you. http://www.codeproject.com/soap/MTOMWebServices.asp my approach is more concerned with providing detailed progress to a winforms user interface. to assist with this, the chunking process is done by code, sending each chunk using MTOM but without the built-in MTOM chunking. the solution is memory efficient and may be helpful for your scenario. hope it helps tim [quoted text, click to view] "smarty" <smarty@nospam.nospam> wrote in message news:83FA17DF-043C-4D10-9DBC-C50E96859C5C@microsoft.com... > Hi, > > I am trying to return a 500Mb file from IIS to a winform client via a web > service. I would like to use MTOM and have installed WSE3 on the client > and > server. I can see there is a chunking setting in the WSE3 settings set to > 64K but I am not sure how I should get the chunking to work correctly. > > I have used some example code for the server setup as shown below but this > seems to involve reading the full 500M into a byte array before returning > it > so WSE can take care of the chunking for me. I can see all the server > memory > being used up when I run the code and the client times out. > > Can you provide some example code and advise to efficiently transfer the > file from the server to the client using chunking without using too much > memory? > > <WebMethod(Description:="This is the MTOM test")> _ > Public Function Test16() As Byte() > Dim response As Byte() > Dim filepath As String = AppDomain.CurrentDomain.BaseDirectory + > "download\" > + filename > response = File.ReadAllBytes(filepath) > Return response > End Function > > Thanks
hi smarty,
that is correct, each chunk is sent as a separate web service call. this makes for easy feedback to the user interface. it also allows for very robust 'resume' functionality, should one of the chunks fail. i remember experimenting with the MTOM built-in chunking and i was not very impressed with it. i can't remember all the reasons why, but i found it better to 'roll my own', especially from the perspective of handling very large files. my code has scaled well for files of several gigabytes. hope this helps tim [quoted text, click to view] "smarty" <smarty@nospam.nospam> wrote in message news:397B0184-E9F4-483E-A984-689E27D8EB07@microsoft.com... > Hi, > > Thanks for that. I have looked at your codeproject article. For > downloading from the server it seems to do it by making succesive web > service > calls. I am after evaluating built in chunking that is specified by the > MTOM > protocol. I am just confused because the WSE3.0 seems to provide settings > for changing chunking but I can't see how to get these to work. The only > examples I have seen populate a byte array with the whole file (storing it > in > memory) and then return the byte array. > > Any ideas? > > "Tim_Mac" wrote: > >> hi smarty >> i wrote an article on codeproject which may be of use to you. >> http://www.codeproject.com/soap/MTOMWebServices.asp >> my approach is more concerned with providing detailed progress to a >> winforms >> user interface. to assist with this, the chunking process is done by >> code, >> sending each chunk using MTOM but without the built-in MTOM chunking. >> the >> solution is memory efficient and may be helpful for your scenario. >> >> hope it helps >> tim >> >> >> "smarty" <smarty@nospam.nospam> wrote in message >> news:83FA17DF-043C-4D10-9DBC-C50E96859C5C@microsoft.com... >> > Hi, >> > >> > I am trying to return a 500Mb file from IIS to a winform client via a >> > web >> > service. I would like to use MTOM and have installed WSE3 on the >> > client >> > and >> > server. I can see there is a chunking setting in the WSE3 settings set >> > to >> > 64K but I am not sure how I should get the chunking to work correctly. >> > >> > I have used some example code for the server setup as shown below but >> > this >> > seems to involve reading the full 500M into a byte array before >> > returning >> > it >> > so WSE can take care of the chunking for me. I can see all the server >> > memory >> > being used up when I run the code and the client times out. >> > >> > Can you provide some example code and advise to efficiently transfer >> > the >> > file from the server to the client using chunking without using too >> > much >> > memory? >> > >> > <WebMethod(Description:="This is the MTOM test")> _ >> > Public Function Test16() As Byte() >> > Dim response As Byte() >> > Dim filepath As String = AppDomain.CurrentDomain.BaseDirectory + >> > "download\" >> > + filename >> > response = File.ReadAllBytes(filepath) >> > Return response >> > End Function >> > >> > Thanks >> >> >>
Hi,
Thanks for that. I have looked at your codeproject article. For downloading from the server it seems to do it by making succesive web service calls. I am after evaluating built in chunking that is specified by the MTOM protocol. I am just confused because the WSE3.0 seems to provide settings for changing chunking but I can't see how to get these to work. The only examples I have seen populate a byte array with the whole file (storing it in memory) and then return the byte array. Any ideas? [quoted text, click to view] "Tim_Mac" wrote:
> hi smarty > i wrote an article on codeproject which may be of use to you. > http://www.codeproject.com/soap/MTOMWebServices.asp > my approach is more concerned with providing detailed progress to a winforms > user interface. to assist with this, the chunking process is done by code, > sending each chunk using MTOM but without the built-in MTOM chunking. the > solution is memory efficient and may be helpful for your scenario. > > hope it helps > tim > > > "smarty" <smarty@nospam.nospam> wrote in message > news:83FA17DF-043C-4D10-9DBC-C50E96859C5C@microsoft.com... > > Hi, > > > > I am trying to return a 500Mb file from IIS to a winform client via a web > > service. I would like to use MTOM and have installed WSE3 on the client > > and > > server. I can see there is a chunking setting in the WSE3 settings set to > > 64K but I am not sure how I should get the chunking to work correctly. > > > > I have used some example code for the server setup as shown below but this > > seems to involve reading the full 500M into a byte array before returning > > it > > so WSE can take care of the chunking for me. I can see all the server > > memory > > being used up when I run the code and the client times out. > > > > Can you provide some example code and advise to efficiently transfer the > > file from the server to the client using chunking without using too much > > memory? > > > > <WebMethod(Description:="This is the MTOM test")> _ > > Public Function Test16() As Byte() > > Dim response As Byte() > > Dim filepath As String = AppDomain.CurrentDomain.BaseDirectory + > > "download\" > > + filename > > response = File.ReadAllBytes(filepath) > > Return response > > End Function > > > > Thanks > >
Hi Tim,
Thanks again for the reply I did like you code and could find it useful for the future but I am trying to evaluate chunking provided with MTOM in .NET and if I get time possilby compare to a Java implementation too. If you have any examples of what you did with the built in MTOM I would be grateful. [quoted text, click to view] "Tim_Mac" wrote:
> hi smarty, > that is correct, each chunk is sent as a separate web service call. this > makes for easy feedback to the user interface. it also allows for very > robust 'resume' functionality, should one of the chunks fail. > i remember experimenting with the MTOM built-in chunking and i was not very > impressed with it. i can't remember all the reasons why, but i found it > better to 'roll my own', especially from the perspective of handling very > large files. my code has scaled well for files of several gigabytes. > > hope this helps > tim > > "smarty" <smarty@nospam.nospam> wrote in message > news:397B0184-E9F4-483E-A984-689E27D8EB07@microsoft.com... > > Hi, > > > > Thanks for that. I have looked at your codeproject article. For > > downloading from the server it seems to do it by making succesive web > > service > > calls. I am after evaluating built in chunking that is specified by the > > MTOM > > protocol. I am just confused because the WSE3.0 seems to provide settings > > for changing chunking but I can't see how to get these to work. The only > > examples I have seen populate a byte array with the whole file (storing it > > in > > memory) and then return the byte array. > > > > Any ideas? > > > > "Tim_Mac" wrote: > > > >> hi smarty > >> i wrote an article on codeproject which may be of use to you. > >> http://www.codeproject.com/soap/MTOMWebServices.asp > >> my approach is more concerned with providing detailed progress to a > >> winforms > >> user interface. to assist with this, the chunking process is done by > >> code, > >> sending each chunk using MTOM but without the built-in MTOM chunking. > >> the > >> solution is memory efficient and may be helpful for your scenario. > >> > >> hope it helps > >> tim > >> > >> > >> "smarty" <smarty@nospam.nospam> wrote in message > >> news:83FA17DF-043C-4D10-9DBC-C50E96859C5C@microsoft.com... > >> > Hi, > >> > > >> > I am trying to return a 500Mb file from IIS to a winform client via a > >> > web > >> > service. I would like to use MTOM and have installed WSE3 on the > >> > client > >> > and > >> > server. I can see there is a chunking setting in the WSE3 settings set > >> > to > >> > 64K but I am not sure how I should get the chunking to work correctly. > >> > > >> > I have used some example code for the server setup as shown below but > >> > this > >> > seems to involve reading the full 500M into a byte array before > >> > returning > >> > it > >> > so WSE can take care of the chunking for me. I can see all the server > >> > memory > >> > being used up when I run the code and the client times out. > >> > > >> > Can you provide some example code and advise to efficiently transfer > >> > the > >> > file from the server to the client using chunking without using too > >> > much > >> > memory? > >> > > >> > <WebMethod(Description:="This is the MTOM test")> _ > >> > Public Function Test16() As Byte() > >> > Dim response As Byte() > >> > Dim filepath As String = AppDomain.CurrentDomain.BaseDirectory + > >> > "download\" > >> > + filename > >> > response = File.ReadAllBytes(filepath) > >> > Return response > >> > End Function > >> > > >> > Thanks > >> > >> > >> > >
hi smarty, sorry i don't have any actual code available. it was mostly
reading from the WSE documentation and looking at the sample applications. good luck tim [quoted text, click to view] "smarty" <smarty@nospam.nospam> wrote in message news:DDB483CC-015B-4BBD-91E3-571CC4A2AB4A@microsoft.com... > Hi Tim, > > Thanks again for the reply I did like you code and could find it useful > for > the future but I am trying to evaluate chunking provided with MTOM in .NET > and if I get time possilby compare to a Java implementation too. > > If you have any examples of what you did with the built in MTOM I would be > grateful. > > "Tim_Mac" wrote: > >> hi smarty, >> that is correct, each chunk is sent as a separate web service call. this >> makes for easy feedback to the user interface. it also allows for very >> robust 'resume' functionality, should one of the chunks fail. >> i remember experimenting with the MTOM built-in chunking and i was not >> very >> impressed with it. i can't remember all the reasons why, but i found it >> better to 'roll my own', especially from the perspective of handling very >> large files. my code has scaled well for files of several gigabytes. >> >> hope this helps >> tim >> >> "smarty" <smarty@nospam.nospam> wrote in message >> news:397B0184-E9F4-483E-A984-689E27D8EB07@microsoft.com... >> > Hi, >> > >> > Thanks for that. I have looked at your codeproject article. For >> > downloading from the server it seems to do it by making succesive web >> > service >> > calls. I am after evaluating built in chunking that is specified by >> > the >> > MTOM >> > protocol. I am just confused because the WSE3.0 seems to provide >> > settings >> > for changing chunking but I can't see how to get these to work. The >> > only >> > examples I have seen populate a byte array with the whole file (storing >> > it >> > in >> > memory) and then return the byte array. >> > >> > Any ideas? >> > >> > "Tim_Mac" wrote: >> > >> >> hi smarty >> >> i wrote an article on codeproject which may be of use to you. >> >> http://www.codeproject.com/soap/MTOMWebServices.asp >> >> my approach is more concerned with providing detailed progress to a >> >> winforms >> >> user interface. to assist with this, the chunking process is done by >> >> code, >> >> sending each chunk using MTOM but without the built-in MTOM chunking. >> >> the >> >> solution is memory efficient and may be helpful for your scenario. >> >> >> >> hope it helps >> >> tim >> >> >> >> >> >> "smarty" <smarty@nospam.nospam> wrote in message >> >> news:83FA17DF-043C-4D10-9DBC-C50E96859C5C@microsoft.com... >> >> > Hi, >> >> > >> >> > I am trying to return a 500Mb file from IIS to a winform client via >> >> > a >> >> > web >> >> > service. I would like to use MTOM and have installed WSE3 on the >> >> > client >> >> > and >> >> > server. I can see there is a chunking setting in the WSE3 settings >> >> > set >> >> > to >> >> > 64K but I am not sure how I should get the chunking to work >> >> > correctly. >> >> > >> >> > I have used some example code for the server setup as shown below >> >> > but >> >> > this >> >> > seems to involve reading the full 500M into a byte array before >> >> > returning >> >> > it >> >> > so WSE can take care of the chunking for me. I can see all the >> >> > server >> >> > memory >> >> > being used up when I run the code and the client times out. >> >> > >> >> > Can you provide some example code and advise to efficiently transfer >> >> > the >> >> > file from the server to the client using chunking without using too >> >> > much >> >> > memory? >> >> > >> >> > <WebMethod(Description:="This is the MTOM test")> _ >> >> > Public Function Test16() As Byte() >> >> > Dim response As Byte() >> >> > Dim filepath As String = AppDomain.CurrentDomain.BaseDirectory + >> >> > "download\" >> >> > + filename >> >> > response = File.ReadAllBytes(filepath) >> >> > Return response >> >> > End Function >> >> > >> >> > Thanks >> >> >> >> >> >> >> >> >>
Thanks for Tim's input.
Hi Smarty, As for the "chunking provided with MTOM" you mentioned, would you provide some further explanation on this? What's the configuration you've seen on this and where did you find the reference about the built-in chunk support of WSE MTOM? Based on my research, the WSE MTOM just use some special encoding rules to compact binary data, however, it will still need to load the complete data into memory and send them through a complete message. What I can find about chunk setting in WSE is internally and not exposed as public setting or interface. I'll discuss with some other WSE engineers to see whether there is a better means to do chunked data transfering, I'll update you as soon as I get any information. Sincerely, Steven Cheng Microsoft MSDN Online Support Lead
Hi Steven,
I was mistaken by the setting for multipart mime thinking this set a chunk value for the streaming. With respect to chunking I was looking at the RRSHB and thought this allowed the message to be chunked, similar to DIME. Looking at the documentation for WSE, there are two examples, one where the whole message is read into a byte array before returning and the other examples utilising the IXMLSerializable interface. The documentation says the streaming support with IXMLSerializable reduces memory and the need to read the information into a byte array before returning. How does this work, does it rely on the client receiving x bytes before the server reads anymore into memory and sends? Thanks [quoted text, click to view] "Steven Cheng[MSFT]" wrote:
> Thanks for Tim's input. > > Hi Smarty, > > As for the "chunking provided with MTOM" you mentioned, would you provide > some further explanation on this? What's the configuration you've seen on > this and where did you find the reference about the built-in chunk support > of WSE MTOM? > > Based on my research, the WSE MTOM just use some special encoding rules to > compact binary data, however, it will still need to load the complete data > into memory and send them through a complete message. What I can find about > chunk setting in WSE is internally and not exposed as public setting or > interface. > > I'll discuss with some other WSE engineers to see whether there is a better > means to do chunked data transfering, I'll update you as soon as I get any > information. > > > Sincerely, > > Steven Cheng > > Microsoft MSDN Online Support Lead > > >
Hi Smarty,
Thanks for your reply. Yes, we does support streaming webservice message through wrapper class that implement IXMLSerializable interface, this is not specific to MTOM data but any request/response object return or pass to webservice webmethods. And I've discussed with some other product team engineers and they've given me some sample code. If you need I can send you through email. You can reach me through the email in my signature(remove "online"). Sincerely, Steven Cheng Microsoft MSDN Online Support Lead
Hi,
I have a couple of questions 1. When trying to use the streaming with large files the client returns a timeout error. I have changed both the file size and timout to -1 in web.config but the error still occurs. After the error has occured I get another error saying the type is invalid. I think this is becuase the previous request is still attempting to stream to the client on the next request. If I leave it for some time the memory on the server is released and I can continue to send smaller files successfully. Any idea how to resolve this? 2. I have seen a note saying WS-Security cannot be applied at an application level when streaming, what is the reason for this, is it due to the DOM parsing? and is this resolved in WCF? Thanks [quoted text, click to view] "Steven Cheng[MSFT]" wrote:
> Hi Smarty, > > Thanks for your reply. > > Yes, we does support streaming webservice message through wrapper class > that implement IXMLSerializable interface, this is not specific to MTOM > data but any request/response object return or pass to webservice > webmethods. And I've discussed with some other product team engineers and > they've given me some sample code. If you need I can send you through > email. You can reach me through the email in my signature(remove "online"). > > > Sincerely, > > Steven Cheng > > Microsoft MSDN Online Support Lead > > >
Hello Smarty, The existing issue I've got from another engineer seems related to ASP.NET webservice that will report a System.Net.WebException after few thousand of calls. This does not crash the service and when caught and handled on the client side the client can start sending out messages again. We have confidence that this issue is outside WSE. Therefore, I think it maybe a bit different from your scenario. BTW, if what you encounter is a timeout issue, you can consider adjust the both the timeout at serverside clientside: **for server-side, in addition to WSE messaging's timeout setting, you can also enlarge ASP.NET <httpruntime> executionTimeout setting. http://msdn2.microsoft.com/en-us/library/e1f13641.aspx ** for client-side, the webservice client proxy class also has a "timeout" property we can configure. If you're still curious about the hotfix or still suffering the timeout issue, I suggest you contact CSS about for further troubleshooting. As far as I know, this kind of message streaming (through the IXmlSerializable wrapper class) is supported. Sincerely, Steven Cheng Microsoft MSDN Online Support Lead
I have found Microsoft's MTOM implementation included in WSE 3.0 to be
useless for large data transfer. The underlying problem is that no matter how you implement your IXMLSerializable, the infrastructure STILL outputs the data into a MemoryStream awaiting transfers to the client (which, of course, may never come). I'm sure that they do this because it was the simplest way to free up the thread filling the MemoryStream buffer. However, the ramification is that you get 'out of memory' exceptions from the server. I was able to reduce the memory footprint somewhat by disabling encryption and signature of the response body. And, of course, you have to completely turn off diagnostic tracing in both the server and the client. Even so, I would regularly run out of memory on a response as small as 300 MB. Sheesh!!! If you want details about the policy that does server-client X.509 authentication without encrypting the response, just ask here or email me. I hate inventing my own wheel when there is a perfectly good specification available (MTOM), but I had to fail back to my own 5 MB chunking algorithm with repeated web service calls in order to avoid Microsft's MTOM implementation in WSE 3.0.
hi tim,
i played with your sample code in http://www.codeproject.com/soap/MTOMWebServices.asp with MTOM. one question: wehre can i see that the file is really transported via MTOM? when i activate the trace function in wse i can't see anything - it's only base64 coded: <inputMessage utc="26.11.2006 14:22:53" messageId="urn:uuid:bf171798-1536-4f00-b2e5-5929177536a7"> <processingStep description="Unprocessed message"> <soap:Envelope xmlns:xop="http://www.w3.org/2004/08/xop/include" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"> <soap:Header> <wsa:Action>http://tim.mackey.ie/CodeProject/MTOM/AppendChunk</wsa:Action> <wsa:MessageID>urn:uuid:bf171798-1536-4f00-b2e5-5929177536a7</wsa:MessageID> <wsa:ReplyTo> <wsa:Address>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</wsa:Address> </wsa:ReplyTo> <wsa:To>http://localhost:1423/WebService/MTOM.asmx</wsa:To> <wsse:Security> <wsu:Timestamp wsu:Id="Timestamp-669ca490-d417-4c0a-a4bd-1ae7f71ec5a2"> <wsu:Created>2006-11-26T14:22:53Z</wsu:Created> <wsu:Expires>2006-11-26T14:27:53Z</wsu:Expires> </wsu:Timestamp> </wsse:Security> </soap:Header> <soap:Body> <AppendChunk xmlns="http://tim.mackey.ie/CodeProject/MTOM"> <FileName>test.kml</FileName> <buffer>PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVR</buffer> <Offset>0</Offset> <BytesRead>16384</BytesRead> </AppendChunk> </soap:Body> </soap:Envelope> </processingStep> or can't you see the MTOM transport in the trace file? greetings from germany stefan "Tim_Mac" <tim.mackey@community.nospam> schrieb im Newsbeitrag news:%23SquVwQAHHA.4844@TK2MSFTNGP02.phx.gbl... [quoted text, click to view] > hi smarty, sorry i don't have any actual code available. it was mostly > reading from the WSE documentation and looking at the sample applications. > > good luck > tim > > > "smarty" <smarty@nospam.nospam> wrote in message > news:DDB483CC-015B-4BBD-91E3-571CC4A2AB4A@microsoft.com... >> Hi Tim, >> >> Thanks again for the reply I did like you code and could find it useful >> for >> the future but I am trying to evaluate chunking provided with MTOM in >> .NET >> and if I get time possilby compare to a Java implementation too. >> >> If you have any examples of what you did with the built in MTOM I would >> be >> grateful. >> >> "Tim_Mac" wrote: >> >>> hi smarty, >>> that is correct, each chunk is sent as a separate web service call. >>> this >>> makes for easy feedback to the user interface. it also allows for very >>> robust 'resume' functionality, should one of the chunks fail. >>> i remember experimenting with the MTOM built-in chunking and i was not >>> very >>> impressed with it. i can't remember all the reasons why, but i found it >>> better to 'roll my own', especially from the perspective of handling >>> very >>> large files. my code has scaled well for files of several gigabytes. >>> >>> hope this helps >>> tim >>> >>> "smarty" <smarty@nospam.nospam> wrote in message >>> news:397B0184-E9F4-483E-A984-689E27D8EB07@microsoft.com... >>> > Hi, >>> > >>> > Thanks for that. I have looked at your codeproject article. For >>> > downloading from the server it seems to do it by making succesive web >>> > service >>> > calls. I am after evaluating built in chunking that is specified by >>> > the >>> > MTOM >>> > protocol. I am just confused because the WSE3.0 seems to provide >>> > settings >>> > for changing chunking but I can't see how to get these to work. The >>> > only >>> > examples I have seen populate a byte array with the whole file >>> > (storing it >>> > in >>> > memory) and then return the byte array. >>> > >>> > Any ideas? >>> > >>> > "Tim_Mac" wrote: >>> > >>> >> hi smarty >>> >> i wrote an article on codeproject which may be of use to you. >>> >> http://www.codeproject.com/soap/MTOMWebServices.asp >>> >> my approach is more concerned with providing detailed progress to a >>> >> winforms >>> >> user interface. to assist with this, the chunking process is done by >>> >> code, >>> >> sending each chunk using MTOM but without the built-in MTOM chunking. >>> >> the >>> >> solution is memory efficient and may be helpful for your scenario. >>> >> >>> >> hope it helps >>> >> tim >>> >> >>> >> >>> >> "smarty" <smarty@nospam.nospam> wrote in message >>> >> news:83FA17DF-043C-4D10-9DBC-C50E96859C5C@microsoft.com... >>> >> > Hi, >>> >> > >>> >> > I am trying to return a 500Mb file from IIS to a winform client via >>> >> > a >>> >> > web >>> >> > service. I would like to use MTOM and have installed WSE3 on the >>> >> > client >>> >> > and >>> >> > server. I can see there is a chunking setting in the WSE3 settings >>> >> > set >>> >> > to >>> >> > 64K but I am not sure how I should get the chunking to work >>> >> > correctly. >>> >> > >>> >> > I have used some example code for the server setup as shown below >>> >> > but >>> >> > this >>> >> > seems to involve reading the full 500M into a byte array before >>> >> > returning >>> >> > it >>> >> > so WSE can take care of the chunking for me. I can see all the >>> >> > server >>> >> > memory >>> >> > being used up when I run the code and the client times out. >>> >> > >>> >> > Can you provide some example code and advise to efficiently >>> >> > transfer >>> >> > the >>> >> > file from the server to the client using chunking without using too >>> >> > much >>> >> > memory? >>> >> > >>> >> > <WebMethod(Description:="This is the MTOM test")> _ >>> >> > Public Function Test16() As Byte() >>> >> > Dim response As Byte() >>> >> > Dim filepath As String = AppDomain.CurrentDomain.BaseDirectory + >>> >> > "download\" >>> >> > + filename >>> >> > response = File.ReadAllBytes(filepath) >>> >> > Return response >>> >> > End Function >>> >> > >>> >> > Thanks >>> >> >>> >> >>> >> >>> >>> >>> > >
Hi,
I have been using the MTOM implementation for streaming and can see an improvment in performance. When using a simple byte array transfering a 100Mb file uses up almost all memory available on my server (800Mb available on perf mon) and I cannot send much larger files. When using the streaming I can see an improvement and have sent files upto 500Mb in size. However the larger the files being sent the larger the memory footprint used. When throughput testing I will receive timeouts if I open more than 1 web service call for a 500Mb file at a time. Is this correct, shouldn't streaming give a similar memory footprint whatever the file size? I am trying to determine if I have implemented this correctly, maybe it isn't streaming correctly? Some example memory footprints for a single web service call with a 100Mb file are... The server memory is 825Mb with no activity. When returning a normal byte array this drops to 320mb and then down to 200mb after a period. When using my streaming implementation (wse3 sample code) the memory drops from an initial 825mb to 650mb. Thanks I will appreciate any help you can give on this. [quoted text, click to view] "Steven Cheng[MSFT]" wrote:
> > Hello Smarty, > > The existing issue I've got from another engineer seems related to ASP.NET > webservice that will report a System.Net.WebException after few thousand of > calls. This does not crash the service and when caught and handled on the > client side the client can start sending out messages again. We have > confidence that this issue is outside WSE. > > Therefore, I think it maybe a bit different from your scenario. BTW, if > what you encounter is a timeout issue, you can consider adjust the both the > timeout at serverside clientside: > > **for server-side, in addition to WSE messaging's timeout setting, you can > also enlarge ASP.NET <httpruntime> executionTimeout setting. > > http://msdn2.microsoft.com/en-us/library/e1f13641.aspx > > ** for client-side, the webservice client proxy class also has a "timeout" > property we can configure. > > If you're still curious about the hotfix or still suffering the timeout > issue, I suggest you contact CSS about for further troubleshooting. As far > as I know, this kind of message streaming (through the IXmlSerializable > wrapper class) is supported. > > > Sincerely, > > Steven Cheng > > Microsoft MSDN Online Support Lead > > >
hi stefan,
sorry i haven't been watching the newsgroups for a while, i just saw your post. my understanding of MTOM is that when it is enabled for a web service on client and server, any web methods that send byte[] as parameters or return types, automatically get handled specially by MTOM, specifically that they are not xml-serialized inside the XML message, as happens on a standard web service, which is obviously very costly for both processing and bandwidth. from my point of view, this embedding of the raw binary data within the message is the most useful feature of MTOM for sending binary data over web services. it is indeed very basic but perhaps you know something i don't? did i miss something? i did a good few tests to compare the performance advantages of an MTOM enabled web service, simply by switching on or off MTOM, and i seem to recall about a 10% performance improvement over DIME and i think about 30% again over plain web services. interested to hear your response. cheers tim [quoted text, click to view] "Stefan Huber" <stefan.huber@b-es.de> wrote in message
news:eUQzEcWEHHA.4620@TK2MSFTNGP04.phx.gbl... > hi tim, > > i played with your sample code in > http://www.codeproject.com/soap/MTOMWebServices.asp with MTOM. > > one question: wehre can i see that the file is really transported via > MTOM? > when i activate the trace function in wse i can't see anything - it's only > base64 coded: > > <inputMessage utc="26.11.2006 14:22:53" > messageId="urn:uuid:bf171798-1536-4f00-b2e5-5929177536a7"> > > <processingStep description="Unprocessed message"> > > <soap:Envelope xmlns:xop="http://www.w3.org/2004/08/xop/include" > xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xmlns:xsd="http://www.w3.org/2001/XMLSchema" > xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" > xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" > xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"> > > <soap:Header> > > <wsa:Action>http://tim.mackey.ie/CodeProject/MTOM/AppendChunk</wsa:Action> > > <wsa:MessageID>urn:uuid:bf171798-1536-4f00-b2e5-5929177536a7</wsa:MessageID> > > <wsa:ReplyTo> > > <wsa:Address>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</wsa:Address> > > </wsa:ReplyTo> > > <wsa:To>http://localhost:1423/WebService/MTOM.asmx</wsa:To> > > <wsse:Security> > > <wsu:Timestamp wsu:Id="Timestamp-669ca490-d417-4c0a-a4bd-1ae7f71ec5a2"> > > <wsu:Created>2006-11-26T14:22:53Z</wsu:Created> > > <wsu:Expires>2006-11-26T14:27:53Z</wsu:Expires> > > </wsu:Timestamp> > > </wsse:Security> > > </soap:Header> > > <soap:Body> > > <AppendChunk xmlns="http://tim.mackey.ie/CodeProject/MTOM"> > > <FileName>test.kml</FileName> > > <buffer>PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVR</buffer> > > <Offset>0</Offset> > > <BytesRead>16384</BytesRead> > > </AppendChunk> > > </soap:Body> > > </soap:Envelope> > > </processingStep> > > > > > > or can't you see the MTOM transport in the trace file? > > greetings from germany > stefan > > > > > > > > > > "Tim_Mac" <tim.mackey@community.nospam> schrieb im Newsbeitrag > news:%23SquVwQAHHA.4844@TK2MSFTNGP02.phx.gbl... >> hi smarty, sorry i don't have any actual code available. it was mostly >> reading from the WSE documentation and looking at the sample >> applications. >> >> good luck >> tim >> >> >> "smarty" <smarty@nospam.nospam> wrote in message >> news:DDB483CC-015B-4BBD-91E3-571CC4A2AB4A@microsoft.com... >>> Hi Tim, >>> >>> Thanks again for the reply I did like you code and could find it useful >>> for >>> the future but I am trying to evaluate chunking provided with MTOM in >>> .NET >>> and if I get time possilby compare to a Java implementation too. >>> >>> If you have any examples of what you did with the built in MTOM I would >>> be >>> grateful. >>> >>> "Tim_Mac" wrote: >>> >>>> hi smarty, >>>> that is correct, each chunk is sent as a separate web service call. >>>> this >>>> makes for easy feedback to the user interface. it also allows for very >>>> robust 'resume' functionality, should one of the chunks fail. >>>> i remember experimenting with the MTOM built-in chunking and i was not >>>> very >>>> impressed with it. i can't remember all the reasons why, but i found >>>> it >>>> better to 'roll my own', especially from the perspective of handling >>>> very >>>> large files. my code has scaled well for files of several gigabytes. >>>> >>>> hope this helps >>>> tim >>>> >>>> "smarty" <smarty@nospam.nospam> wrote in message >>>> news:397B0184-E9F4-483E-A984-689E27D8EB07@microsoft.com... >>>> > Hi, >>>> > >>>> > Thanks for that. I have looked at your codeproject article. For >>>> > downloading from the server it seems to do it by making succesive web >>>> > service >>>> > calls. I am after evaluating built in chunking that is specified by >>>> > the >>>> > MTOM >>>> > protocol. I am just confused because the WSE3.0 seems to provide >>>> > settings >>>> > for changing chunking but I can't see how to get these to work. The >>>> > only >>>> > examples I have seen populate a byte array with the whole file >>>> > (storing it >>>> > in >>>> > memory) and then return the byte array. >>>> > >>>> > Any ideas? >>>> > >>>> > "Tim_Mac" wrote: >>>> > >>>> >> hi smarty >>>> >> i wrote an article on codeproject which may be of use to you. >>>> >> http://www.codeproject.com/soap/MTOMWebServices.asp >>>> >> my approach is more concerned with providing detailed progress to a >>>> >> winforms >>>> >> user interface. to assist with this, the chunking process is done by >>>> >> code, >>>> >> sending each chunk using MTOM but without the built-in MTOM >>>> >> chunking. >>>> >> the >>>> >> solution is memory efficient and may be helpful for your scenario. >>>> >> >>>> >> hope it helps >>>> >> tim >>>> >> >>>> >> >>>> >> "smarty" <smarty@nospam.nospam> wrote in message >>>> >> news:83FA17DF-043C-4D10-9DBC-C50E96859C5C@microsoft.com... >>>> >> > Hi, >>>> >> > >>>> >> > I am trying to return a 500Mb file from IIS to a winform client >>>> >> > via a >>>> >> > web >>>> >> > service. I would like to use MTOM and have installed WSE3 on the >>>> >> > client >>>> >> > and >>>> >> > server. I can see there is a chunking setting in the WSE3 >>>> >> > settings set >>>> >> > to >>>> >> > 64K but I am not sure how I should get the chunking to work >>>> >> > correctly. >>>> >> > >>>> >> > I have used some example code for the server setup as shown below >>>> >> > but >>>> >> > this >>>> >> > seems to involve reading the full 500M into a byte array before >>>> >> > returning >>>> >> > it >>>> >> > so WSE can take care of the chunking for me. I can see all the >>>> >> > server
Hi Smarty,
I think this should be due to the WSE MTOM's natural limitation, it still need to load block of data into memory when transfering large data through streaming, it is not unlimited. Have you ever contact the product support team on this issue and find any result on this? Sincerely, Steven Cheng Microsoft MSDN Online Support Lead
Don't see what you're looking for? Try a search.
|
June 2004
July 2004
August 2004
September 2004
October 2004
November 2004
December 2004
January 2005
February 2005
March 2005
April 2005
May 2005
June 2005
July 2005
August 2005
September 2005
October 2005
November 2005
December 2005
January 2006
February 2006
March 2006
April 2006
May 2006
June 2006
July 2006
August 2006
September 2006
October 2006
November 2006
December 2006
January 2007
February 2007
March 2007
April 2007
May 2007
June 2007
July 2007
August 2007
September 2007
October 2007
November 2007
December 2007
March 2008
April 2008
May 2008
June 2008
| home · blog · groups | Questions? Comments? Contact the dn |