|
|
You're in the
dotnet clr
group:copying stream - advise needed
dotnet clr:
hi,
Iv'e seen two ways of copying streams in .net: Stream outStream = MemoryStream(); Stream inStream = CreateSomeMemoryStream(); byte[] buffer = new byte[inStream.Length]; inStream.Read(buffer, 0, buffer.Length); outStream.Write(buffer, 0, buffer.Length); and the second one was more or less like this: Stream outStream = MemoryStream(); Stream inStream = CreateSomeMemoryStream(); int size = 2048; byte[] buffer = new byte[size]; while (true) { size = inStream.Read(buffer, 0, buffer.Length); if (size > 0) outStream.Write(buffer, 0, size); else break; } i wonder if there are any guidelines on this issue? should i read in a
[quoted text, click to view] SharpCoderMP wrote: > hi, > > Iv'e seen two ways of copying streams in .net: > > Stream outStream = MemoryStream(); > Stream inStream = CreateSomeMemoryStream(); I assume your're not always reading and writing to memory-streams. [quoted text, click to view] > byte[] buffer = new byte[inStream.Length]; > inStream.Read(buffer, 0, buffer.Length); > outStream.Write(buffer, 0, buffer.Length); That's just plain horrible, it only works on streams with known length, and it isn't ensured to read the entire in-stream (in the general case streams may not return as much data as asked for, even it is available). [quoted text, click to view] > and the second one was more or less like this: > > Stream outStream = MemoryStream(); > Stream inStream = CreateSomeMemoryStream(); I still don't understand these memory-streams..., do you really *need* a memory-copy of the content of inStream? [quoted text, click to view] > int size = 2048; > byte[] buffer = new byte[size]; > while (true) > { > size = inStream.Read(buffer, 0, buffer.Length); > if (size > 0) > outStream.Write(buffer, 0, size); > else > break; > } This is quite a lot better. I prefer a do...while loop though. [quoted text, click to view] > > i wonder if there are any guidelines on this issue? should i read in a > loop or can i read at once and the CLR will take care of everything? You *should* read in a loop, since Read may not return the entire content of the stream. Here is how I usually do it: using System.IO; class StreamUtil { public static void CopyStream( Stream input, Stream output, byte[] buffer, int offset, int count) { int read; do { read = input.Read(buffer, offset, count); if ( read > 0 ) output.Write(buffer, offset, read); } while ( read > 0 ); } public static void CopyStream(Stream input, Stream output, byte[] buffer) { CopyStream(input, output, buffer, 0, buffer.Length); } public static void CopyStream(Stream input, Stream output, int bufferSize) { CopyStream(input, output, new byte[bufferSize]); } public static void CopyStream(Stream input, Stream output) { CopyStream(input, output, 8*1024); } // ... more utils here } class Foo { void DoingStuff() { // ... Stream input; // from somewhere Stream output; // from somewhere StreamUtil.CopyStream(input, output); // ... } } -- Helge Jensen mailto:helge.jensen@slog.dk sip:helge.jensen@slog.dk
[quoted text, click to view]
>i wonder if there are any guidelines on this issue? should i read in a >loop or can i read at once and the CLR will take care of everything? IMO you should use a loop. While it may not be needed when working exclusively with MemoryStreams, it could be necessary for other type of Streams. That doesn't mean you have to read 2K at the time though. If the input stream has a known length you can directly create a byte[] of the correct size and keep reading into that. You can then construct the output stream around that byte array with the MemoryStream(byte[]) constructor to avoid making a third copy of the data. Mattias -- Mattias Sjögren [C# MVP] mattias @ mvps.org http://www.msjogren.net/dotnet/ | http://www.dotnetinterop.com
One thing that has yet to be mentioned as a reason why its better to use the
looping methodology is large objects ... if you use the simple read/write methodology you end up with a byte array the size of the data being created. If this byte size is over a certain limit (I forget what the exact number is off the top of my head but want to say 85k ??? someone feel free to correct me) it is treated as a large object and put on a special heap. Large objects have differing life cycles and are not garbage collected in the same way as normal objects .. this can lead to a possible memory usage attack by someone forcing this operation to happen repeatedly ... I have never tried to take an app down in this way but I would imagine that it is feasible. Cheers, Greg Young [quoted text, click to view] "SharpCoderMP" <csharp_mp@interia.pl.NFSPM> wrote in message news:%23Cfl4JFaGHA.1560@TK2MSFTNGP03.phx.gbl... > hi, > > Iv'e seen two ways of copying streams in .net: > > Stream outStream = MemoryStream(); > Stream inStream = CreateSomeMemoryStream(); > byte[] buffer = new byte[inStream.Length]; > inStream.Read(buffer, 0, buffer.Length); > outStream.Write(buffer, 0, buffer.Length); > > and the second one was more or less like this: > > Stream outStream = MemoryStream(); > Stream inStream = CreateSomeMemoryStream(); > int size = 2048; > byte[] buffer = new byte[size]; > while (true) > { > size = inStream.Read(buffer, 0, buffer.Length); > if (size > 0) > outStream.Write(buffer, 0, size); > else > break; > } > > i wonder if there are any guidelines on this issue? should i read in a > loop or can i read at once and the CLR will take care of everything?
[quoted text, click to view]
Helge Jensen wrote: yes, you're right.> I assume your're not always reading and writing to memory-streams. [quoted text, click to view] > I still don't understand these memory-streams..., do you really *need* a this was rather for the purpose of this example. usually i work with> memory-copy of the content of inStream? streams produced by some classes or created from files and i cannot predict their origin. i usually end up with something like this: Stream myStream = SomeThirdPartyClass.GetStream(); and then i need to do some processing before i can use this stream. the processing does some major changes to the data inside the stream changing also it's length. If i would like to then write myStream to file i need to create new FileStream and copy the contents of the myStream into that newly created FileStream. [quoted text, click to view] > This is quite a lot better. I prefer a do...while loop though. yeah i know.[quoted text, click to view] >> i wonder if there are any guidelines on this issue? should i read in a ok. thanks for the info.>> loop or can i read at once and the CLR will take care of everything? > > You *should* read in a loop, since Read may not return the entire > content of the stream. [quoted text, click to view] >
> Here is how I usually do it: > > using System.IO; > class StreamUtil > { > public static void CopyStream( > Stream input, Stream output, > byte[] buffer, int offset, int count) > { > int read; > do > { > read = input.Read(buffer, offset, count); > if ( read > 0 ) > output.Write(buffer, offset, read); > } while ( read > 0 ); > } > public static void CopyStream(Stream input, Stream output, byte[] buffer) > { CopyStream(input, output, buffer, 0, buffer.Length); } > public static void CopyStream(Stream input, Stream output, int bufferSize) > { CopyStream(input, output, new byte[bufferSize]); } > public static void CopyStream(Stream input, Stream output) > { CopyStream(input, output, 8*1024); } > // ... more utils here > } > class Foo > { > void DoingStuff() { > // ... > Stream input; // from somewhere > Stream output; // from somewhere > StreamUtil.CopyStream(input, output); > // ... > } > }
[quoted text, click to view] Greg Young [MVP] wrote: > One thing that has yet to be mentioned as a reason why its better to use the > looping methodology is large objects ... The looping strategy is required for semantic reasons. A stream may return less bytes than requested, even if the stream is not closed yet. invoking s.Read(buf, 0, bug.Length) may return at any time there is any data available, or the stream is closed. In the real world not-looping will often break with for example network-streams, where data arrive "slowly". [quoted text, click to view] > if you use the simple read/write methodology you end up with a byte array > the size of the data being created. If this byte size is over a certain > limit (I forget what the exact number is off the top of my head but want to > say 85k ??? someone feel free to correct me) it is treated as a large object > and put on a special heap. Large objects have differing life cycles and are > not garbage collected in the same way as normal objects .. this can lead to > a possible memory usage attack by someone forcing this operation to happen Creating a large amount of large objects is also not good, but it's not semanticly required. [quoted text, click to view] > repeatedly ... I have never tried to take an app down in this way but I > would imagine that it is feasible. That can most certainly be done :) I have seen code (in effect) like: while ( true ) { using ( MemoryStream s = new MemoryStream() ) using ( Stream connection = awaitConnectionAndGetStream() ) { // workaround Deserialize invoke s.Close() Util.StreamCopy(connection, s); try { object o = formatter.Deserialize(s); } /// process and send reply } } Conversing with this server for any significant amount of time, transmitting a few objects of 85k the .NET runtime would break down with an out-of-memory error. If you expect (or risk receiving) large amounts of input, you should process the data from that stream in a streaming manner, that is -- a "little" at a time. In the end i rewrote the above-code to use pass a proxy for the connection to Deserialize. That proxy workarounds the Close, and some other issues and it allows the caller to limit the amount of data that can be read from the proxy. -- Helge Jensen mailto:helge.jensen@slog.dk sip:helge.jensen@slog.dk
I understand it is required. I was listing the large objects as an
additional benefit :) "> One thing that has yet to be mentioned as a reason why its better to use the [quoted text, click to view] > looping methodology is large objects ..." That is to say "in addition to" the other reasons mentioned. Cheers, Greg [quoted text, click to view] "Helge Jensen" <helge.jensen@slog.dk> wrote in message news:eMtqaWQaGHA.3444@TK2MSFTNGP05.phx.gbl... > > > Greg Young [MVP] wrote: >> One thing that has yet to be mentioned as a reason why its better to use >> the >> looping methodology is large objects ... > > The looping strategy is required for semantic reasons. A stream may > return less bytes than requested, even if the stream is not closed yet. > > invoking s.Read(buf, 0, bug.Length) may return at any time there is any > data available, or the stream is closed. > > In the real world not-looping will often break with for example > network-streams, where data arrive "slowly". > >> if you use the simple read/write methodology you end up with a byte array >> the size of the data being created. If this byte size is over a certain >> limit (I forget what the exact number is off the top of my head but want >> to >> say 85k ??? someone feel free to correct me) it is treated as a large >> object >> and put on a special heap. Large objects have differing life cycles and >> are >> not garbage collected in the same way as normal objects .. this can lead >> to >> a possible memory usage attack by someone forcing this operation to >> happen > > Creating a large amount of large objects is also not good, but it's not > semanticly required. > >> repeatedly ... I have never tried to take an app down in this way but I >> would imagine that it is feasible. > > That can most certainly be done :) I have seen code (in effect) like: > > while ( true ) { > using ( MemoryStream s = new MemoryStream() ) > using ( Stream connection = awaitConnectionAndGetStream() ) { > // workaround Deserialize invoke s.Close() > Util.StreamCopy(connection, s); > try { > object o = formatter.Deserialize(s); > } > /// process and send reply > } > } > > Conversing with this server for any significant amount of time, > transmitting a few objects of 85k the .NET runtime would break down with > an out-of-memory error. > > If you expect (or risk receiving) large amounts of input, you should > process the data from that stream in a streaming manner, that is -- a > "little" at a time. > > In the end i rewrote the above-code to use pass a proxy for the > connection to Deserialize. That proxy workarounds the Close, and some > other issues and it allows the caller to limit the amount of data that > can be read from the proxy. > > -- > Helge Jensen > mailto:helge.jensen@slog.dk > sip:helge.jensen@slog.dk > -=> Sebastian cover-music: http://ungdomshus.nu <=-
[quoted text, click to view] Greg Young [MVP] wrote: > I understand it is required. I was listing the large objects as an > additional benefit :) Okay, I see now. <splitting hairs> [quoted text, click to view] > "> One thing that has yet to be mentioned as a reason why its better to use > the > >>looping methodology is large objects ..." > > > That is to say "in addition to" the other reasons mentioned. What happened was; I don't really see that there is any choice in the matter, looping is required for correctness, not for optimization. That made me choke a bit on the "conditional" way your reply was formulated: "why *it's better* to use the looping..." [My emphasis] It would have went down a treat as something along the lines of: "In addition to the semantics/correcteness issues of not-looping, there is the issue of large objects ..." </splitting hairs> -- Helge Jensen mailto:helge.jensen@slog.dk sip:helge.jensen@slog.dk
Don't see what you're looking for? Try a search.
|
June 2003
July 2003
August 2003
September 2003
October 2003
November 2003
December 2003
January 2004
February 2004
March 2004
April 2004
May 2004
June 2004
July 2004
August 2004
September 2004
October 2004
November 2004
December 2004
January 2005
February 2005
March 2005
April 2005
May 2005
June 2005
July 2005
August 2005
September 2005
October 2005
November 2005
December 2005
January 2006
February 2006
March 2006
April 2006
May 2006
June 2006
July 2006
August 2006
September 2006
October 2006
November 2006
December 2006
January 2007
February 2007
March 2007
April 2007
May 2007
June 2007
July 2007
August 2007
September 2007
October 2007
November 2007
December 2007
January 2008
February 2008
March 2008
April 2008
May 2008
June 2008
| home · blog · groups | Questions? Comments? Contact the dn |