Groups | Blog | Home
all groups > dotnet web services enhancements > june 2006 >

dotnet web services enhancements : Transferring Large (GB+ size) files with MTOM


Chris Mullins
6/7/2006 11:22:18 AM
I need to build an infrastructure that moves large files around the
Internet. The scenario is many clients all submitting information to a
single server. The data in question is confidential, so encryption and
signatures are important. Customers currently write data to magnetic tape
and have that tape delivered via courrier.

The files in question will range from 10 megs to about 50 gigs, and we may
see (on a very rare occasion) as many as 10,000 files in a single day. More
normal load will be 1000 or fewer files per day. Bandwith isn't an issue,
and there are lots of machines available in a web farm.

As each client is likley to want to develop their own client side
implementation (so they can seamlessly integrate into their existing
business processes), it's important that the protocols used be as standard
as possible. Everyone's first impression was FTP until they realized it
doesn't meet the security constraints.

I've been looking at MTOM (via WSE 3.0) and trying to decide if it's going
to do the trick. So far, it doesn't look as if MTOM can deal well with large
files, and everything I've seen really requires a chunking approach. While I
can live with requiring chunking, I would rather have a solution that didn't
require this.

I am, I must confess, disappointed that MTOM doesn't seem to cope at all
with large files. From what I can see in WSE, the entire MTOM image is
stored in memory until the whole thing is received, then is written out to
disk. I need something that is a bit more of a streaming approach. Am I
doing something wrong? I'm open to suggestions...

--
Chris Mullins

olly
6/8/2006 2:39:01 AM
Wouldnt you be better using a know protocol such as SFTP, part of the SSH
tool? Versions are available for pretty much every platform, and at least the
security implications of the protocol are well understood. I am sure there
must be plenty of secure file transfer tools out there without having to
reinvent the wheel. Also established protocols are probably optimised rather
then sending the whole file in a single soap request. Also they support
resuming upload if the connection was reset.

Hope that helps,

Olly

[quoted text, click to view]
AddThis Social Bookmark Button