|
|
You're in the
dotnet distributed apps
group:Updating DB with large collections
dotnet distributed apps:
My application is composed of business entities which are in the form of classes instances. Some of these entities contains large arraylists of other entities. These entities are being sent to the client - where the user might add/modify/delete some items in the arraylists. Natuaraly, I don't want to send to the DB the part of the entities which the user hadn't changed. I'm keeping a copy of the original entity in my middle tier so when the modified entity is returned from the client I can check what was changed and according to this decide what to send to my data layer. The question is what would be the best method to do the comparison? I was advised to use dataset. I've tried to serialize my arraylists - both of the original object and the modified one - into datasets and use the merge method. But I couldn't get it to work (note that the client is not getting a dataset but rather a class instance). Is there a way to make it work following the above scenario? If there isn't what would be the most appropriate way to perform that kind of operation? Thanks,
Sounds like you might simply have to write compare methods for each
object or override the Equals method for each object. You might want to simply update all properties irregardless for performance reasons. Comparing lots of objects properties can be a significant overhead; outweigh the costs of simply updating all data fields. You can keep a "dirty" flag for each object, so that you only update changed objects, but not worry about each element change. You could also develop a base class and a small hierarchy of derived classes (to represent Strings, Integers, etc.)for each property/element that would have a "dirty" flag as well. Then you could easily determine which properties have changed for each class. You might want to review these articles and guides: ..NET Data Access Architecture Guide http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/daag.asp and Passing Data Between Objects in an Application http://msdn.microsoft.com/msdnmag/issues/03/07/AdvancedBasics/default.aspx Thanks, Sam -- _______________________________ Sam Santiago ssantiago@n0spam-SoftiTechture.com http://www.SoftiTechture.com _______________________________ [quoted text, click to view] "gwenda" <gwenda@discussions.microsoft.com> wrote in message news:ADC89C9E-2399-4FF4-9566-10261B0FCD24@microsoft.com... > Hi, > My application is composed of business entities which are in the form of > classes instances. Some of these entities contains large arraylists of other > entities. These entities are being sent to the client - where the user might > add/modify/delete some items in the arraylists. Natuaraly, I don't want to > send to the DB the part of the entities which the user hadn't changed. > I'm keeping a copy of the original entity in my middle tier so when the > modified entity is returned from the client I can check what was changed and > according to this decide what to send to my data layer. > The question is what would be the best method to do the comparison? > I was advised to use dataset. I've tried to serialize my arraylists - both > of the original object and the modified one - into datasets and use the merge > method. But I couldn't get it to work (note that the client is not getting a > dataset but rather a class instance). Is there a way to make it work > following the above scenario? If there isn't what would be the most > appropriate way to perform that kind of operation? > Thanks, > Gwenda
Hi Sam,
Thanks for your answer. There are few points I would like to point out regarding your answer: Generally, I intend to update all the properties of any object. However, I don’t want to update (if I don’t have to) two types of properties: Collections –usually contains other objects. I would like to insert/update/delete only the objects in the collection that were modified by the client. Some of my objects contain rather large collections and some may contain more than one property which is a collection. Large sub-objects – that is an object which is contained in the main object and is large enough to reconsider updating it for no reason. Performance is indeed a consideration but another motive was the DB concurrency. Naturally those sub-objects and collections usually reside in other tables in the DB and may have been modified but other users. Although I have a mechanism to prevent it, I’ve learned that those big updates which wipe everything an re-insert new date usually turn to be messy. Although the “dirty†flag is a possibility, it is a “dirty†one… :-) And in case of external systems/other UI that will update my system, I will have to make sure that the flags were used. I was thinking about having my business component in the middle tier comparing the cached object with the modified object. According to the comparison results the relevant main-object and/or sub-objects and/or collections objects will be sent to the data layer components. The collections can be serialized to XML strings and sent to the DB as such. But this seems like a complex solution to what I imagined should be quite common problem and I kind of hoped for a “by-the-book†solution. Thanks Again, Gwenda [quoted text, click to view] "gwenda" wrote:
> Hi, > My application is composed of business entities which are in the form of > classes instances. Some of these entities contains large arraylists of other > entities. These entities are being sent to the client - where the user might > add/modify/delete some items in the arraylists. Natuaraly, I don't want to > send to the DB the part of the entities which the user hadn't changed. > I'm keeping a copy of the original entity in my middle tier so when the > modified entity is returned from the client I can check what was changed and > according to this decide what to send to my data layer. > The question is what would be the best method to do the comparison? > I was advised to use dataset. I've tried to serialize my arraylists - both > of the original object and the modified one - into datasets and use the merge > method. But I couldn't get it to work (note that the client is not getting a > dataset but rather a class instance). Is there a way to make it work > following the above scenario? If there isn't what would be the most > appropriate way to perform that kind of operation? > Thanks,
[quoted text, click to view]
> Although the "dirty" flag is a possibility, it is a "dirty" one. :-) And in > case of external systems/other UI that will update my system, I will have to > make sure that the flags were used. The "dirty" flag would not be visible to any external systems or clients of your objects. The flag would purely be an internally managed state mechanism. [quoted text, click to view] > The collections can be serialized to XML strings and sent to the DB as such.Are you saving the entire collection in the DB as an XML string? I thought these collections were discrete objects where each object would have to be saved in the DB if modified. Concurrency is always an issue and should be dealt with by the data layer - never update the DB if the object to be saved has stale data. If you have dealt with concurrency properly, then whether or not you update all fields in a row or just a subset of fields in a row becomes purely a performance issue and not a data integrity issue. Have you seen this application block on MSDN: Data Access Application Block http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/daab-rm.asp Thanks, Sam -- _______________________________ Sam Santiago ssantiago@n0spam-SoftiTechture.com http://www.SoftiTechture.com _______________________________ [quoted text, click to view] "gwenda" <gwenda@discussions.microsoft.com> wrote in message news:8DA59B37-5648-45EE-B875-18B2BA3FB552@microsoft.com... > Hi Sam, > > Thanks for your answer. There are few points I would like to point out > regarding your answer: > > Generally, I intend to update all the properties of any object. However, I > don't want to update (if I don't have to) two types of properties: > > Collections -usually contains other objects. I would like to > insert/update/delete only the objects in the collection that were modified by > the client. Some of my objects contain rather large collections and some may > contain more than one property which is a collection. > > Large sub-objects - that is an object which is contained in the main object > and is large enough to reconsider updating it for no reason. > > Performance is indeed a consideration but another motive was the DB > concurrency. Naturally those sub-objects and collections usually reside in > other tables in the DB and may have been modified but other users. Although I > have a mechanism to prevent it, I've learned that those big updates which > wipe everything an re-insert new date usually turn to be messy. > > Although the "dirty" flag is a possibility, it is a "dirty" one. :-) And in > case of external systems/other UI that will update my system, I will have to > make sure that the flags were used. > > I was thinking about having my business component in the middle tier > comparing the cached object with the modified object. According to the > comparison results the relevant main-object and/or sub-objects and/or > collections objects will be sent to the data layer components. The > collections can be serialized to XML strings and sent to the DB as such. But > this seems like a complex solution to what I imagined should be quite common > problem and I kind of hoped for a "by-the-book" solution. > > Thanks Again, > > Gwenda > > > > > > "gwenda" wrote: > > > Hi, > > My application is composed of business entities which are in the form of > > classes instances. Some of these entities contains large arraylists of other > > entities. These entities are being sent to the client - where the user might > > add/modify/delete some items in the arraylists. Natuaraly, I don't want to > > send to the DB the part of the entities which the user hadn't changed. > > I'm keeping a copy of the original entity in my middle tier so when the > > modified entity is returned from the client I can check what was changed and > > according to this decide what to send to my data layer. > > The question is what would be the best method to do the comparison? > > I was advised to use dataset. I've tried to serialize my arraylists - both > > of the original object and the modified one - into datasets and use the merge > > method. But I couldn't get it to work (note that the client is not getting a > > dataset but rather a class instance). Is there a way to make it work > > following the above scenario? If there isn't what would be the most > > appropriate way to perform that kind of operation? > > Thanks, > > Gwenda
Hi Sam,
[quoted text, click to view] > The "dirty" flag would not be visible to any external systems or clients of > your objects. The flag would purely be an internally managed state > mechanism. I might have failed to understand your solution. But what I mean by external systems is that I can’t expect those to use this flag and so when my system is updated externally I will be forced to update the whole object – regardless of the changes made in it. With other clients I will have to make sure each is using this flag correctly. [quoted text, click to view] > Are you saving the entire collection in the DB as an XML string? I thought > these collections were discrete objects where each object would have to be > saved in the DB if modified. I don’t keep them as xml in the DB. I intend to use SQL server 2000 sp_xml_preparedocument to convert the string back to XML and then use the OPENXML command and a temporary table to read the data from the xml and update whatever needed to be updated. [quoted text, click to view] >Concurrency is always an issue and should be > dealt with by the data layer - never update the DB if the object to be saved > has stale data. If you have dealt with concurrency properly, then whether > or not you update all fields in a row or just a subset of fields in a row > becomes purely a performance issue and not a data integrity issue. Sorry. I use the term concurrency too broadly. What I meant is that wiping all current data in favor of the new data chunk have caused me, in the past, to lose data – especially in scenarios of business rules which rely on certain date-time data. I’m sure it could have been avoided but it still feels wrong to do so. [quoted text, click to view] > Have you seen this application block on MSDN: > > Data Access Application Block > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/daab-rm.asp Sure – I’ve worked before with its former version and intend to use it in the current solution too. As far as I recall, this block does not provide answers to my question – or am I wrong? Thanks, Gwenda [quoted text, click to view] "Sam Santiago" wrote:
> > Although the "dirty" flag is a possibility, it is a "dirty" one. :-) And > in > > case of external systems/other UI that will update my system, I will have > to > > make sure that the flags were used. > > The "dirty" flag would not be visible to any external systems or clients of > your objects. The flag would purely be an internally managed state > mechanism. > > > The collections can be serialized to XML strings and sent to the DB as > such. > > Are you saving the entire collection in the DB as an XML string? I thought > these collections were discrete objects where each object would have to be > saved in the DB if modified. Concurrency is always an issue and should be > dealt with by the data layer - never update the DB if the object to be saved > has stale data. If you have dealt with concurrency properly, then whether > or not you update all fields in a row or just a subset of fields in a row > becomes purely a performance issue and not a data integrity issue. > > > Thanks, > > Sam > > > -- > _______________________________ > Sam Santiago > ssantiago@n0spam-SoftiTechture.com > http://www.SoftiTechture.com > _______________________________ > "gwenda" <gwenda@discussions.microsoft.com> wrote in message > news:8DA59B37-5648-45EE-B875-18B2BA3FB552@microsoft.com... > > Hi Sam, > > > > Thanks for your answer. There are few points I would like to point out > > regarding your answer: > > > > Generally, I intend to update all the properties of any object. However, I > > don't want to update (if I don't have to) two types of properties: > > > > Collections -usually contains other objects. I would like to > > insert/update/delete only the objects in the collection that were modified > by > > the client. Some of my objects contain rather large collections and some > may > > contain more than one property which is a collection. > > > > Large sub-objects - that is an object which is contained in the main > object > > and is large enough to reconsider updating it for no reason. > > > > Performance is indeed a consideration but another motive was the DB > > concurrency. Naturally those sub-objects and collections usually reside in > > other tables in the DB and may have been modified but other users. > Although I > > have a mechanism to prevent it, I've learned that those big updates which > > wipe everything an re-insert new date usually turn to be messy. > > > > Although the "dirty" flag is a possibility, it is a "dirty" one. :-) And > in > > case of external systems/other UI that will update my system, I will have > to > > make sure that the flags were used. > > > > I was thinking about having my business component in the middle tier > > comparing the cached object with the modified object. According to the > > comparison results the relevant main-object and/or sub-objects and/or > > collections objects will be sent to the data layer components. The > > collections can be serialized to XML strings and sent to the DB as such. > But > > this seems like a complex solution to what I imagined should be quite > common > > problem and I kind of hoped for a "by-the-book" solution. > > > > Thanks Again, > > > > Gwenda > > > > > > > > > > > > "gwenda" wrote: > > > > > Hi, > > > My application is composed of business entities which are in the form of > > > classes instances. Some of these entities contains large arraylists of > other > > > entities. These entities are being sent to the client - where the user > might > > > add/modify/delete some items in the arraylists. Natuaraly, I don't want > to > > > send to the DB the part of the entities which the user hadn't changed. > > > I'm keeping a copy of the original entity in my middle tier so when the > > > modified entity is returned from the client I can check what was changed > and > > > according to this decide what to send to my data layer. > > > The question is what would be the best method to do the comparison? > > > I was advised to use dataset. I've tried to serialize my arraylists - > both > > > of the original object and the modified one - into datasets and use the > merge > > > method. But I couldn't get it to work (note that the client is not > getting a > > > dataset but rather a class instance). Is there a way to make it work > > > following the above scenario? If there isn't what would be the most > > > appropriate way to perform that kind of operation? > > > Thanks, > > > Gwenda > >
[quoted text, click to view]
> I might have failed to understand your solution. But what I mean by external > systems is that I can't expect those to use this flag and so when my system > is updated externally I will be forced to update the whole object - > regardless of the changes made in it. With other clients I will have to make > sure each is using this flag correctly. The dirty flag would not be exposed to the outside word and clients would not know of it's presence. Your object manages it internally - data encapsulation. For example, whenever a property is changed on your object it would not only change the value of the property, but also set the dirty flag to true. Didn't mean to go off on a tangent, but just wanted to give you other methods that might provide you with further ideas. [quoted text, click to view] > Sure - I've worked before with its former version and intend to use it in > the current solution too. As far as I recall, this block does not provide > answers to my question - or am I wrong? Just wanted to ensure you were aware of its existence. It might not directly answer your question, but if you are familiar with how it deals with passing data through the layers it could potentially also give you further ideas. Good luck. Thanks, Sam -- _______________________________ Sam Santiago ssantiago@n0spam-SoftiTechture.com http://www.SoftiTechture.com _______________________________ [quoted text, click to view] "gwenda" <gwenda@discussions.microsoft.com> wrote in message
news:69BCE50C-32FC-4E7C-8A23-67A3BEA65A8C@microsoft.com... > Hi Sam, > > The "dirty" flag would not be visible to any external systems or clients of > > your objects. The flag would purely be an internally managed state > > mechanism. > > I might have failed to understand your solution. But what I mean by external > systems is that I can't expect those to use this flag and so when my system > is updated externally I will be forced to update the whole object - > regardless of the changes made in it. With other clients I will have to make > sure each is using this flag correctly. > > > Are you saving the entire collection in the DB as an XML string? I thought > > these collections were discrete objects where each object would have to be > > saved in the DB if modified. > > I don't keep them as xml in the DB. I intend to use SQL server 2000 > sp_xml_preparedocument to convert the string back to XML and then use the > OPENXML command and a temporary table to read the data from the xml and > update whatever needed to be updated. > > >Concurrency is always an issue and should be > > dealt with by the data layer - never update the DB if the object to be saved > > has stale data. If you have dealt with concurrency properly, then whether > > or not you update all fields in a row or just a subset of fields in a row > > becomes purely a performance issue and not a data integrity issue. > > Sorry. I use the term concurrency too broadly. What I meant is that wiping > all current data in favor of the new data chunk have caused me, in the past, > to lose data - especially in scenarios of business rules which rely on > certain date-time data. I'm sure it could have been avoided but it still > feels wrong to do so. > > > Have you seen this application block on MSDN: > > > > Data Access Application Block > > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/daab-rm.asp > > Sure - I've worked before with its former version and intend to use it in > the current solution too. As far as I recall, this block does not provide > answers to my question - or am I wrong? > > > Thanks, > Gwenda > > "Sam Santiago" wrote: > > > > Although the "dirty" flag is a possibility, it is a "dirty" one. :-) And > > in > > > case of external systems/other UI that will update my system, I will have > > to > > > make sure that the flags were used. > > > > The "dirty" flag would not be visible to any external systems or clients of > > your objects. The flag would purely be an internally managed state > > mechanism. > > > > > The collections can be serialized to XML strings and sent to the DB as > > such. > > > > Are you saving the entire collection in the DB as an XML string? I thought > > these collections were discrete objects where each object would have to be > > saved in the DB if modified. Concurrency is always an issue and should be > > dealt with by the data layer - never update the DB if the object to be saved > > has stale data. If you have dealt with concurrency properly, then whether > > or not you update all fields in a row or just a subset of fields in a row > > becomes purely a performance issue and not a data integrity issue. > > > > > > Thanks, > > > > Sam > > > > > > -- > > _______________________________ > > Sam Santiago > > ssantiago@n0spam-SoftiTechture.com > > http://www.SoftiTechture.com > > _______________________________ > > "gwenda" <gwenda@discussions.microsoft.com> wrote in message > > news:8DA59B37-5648-45EE-B875-18B2BA3FB552@microsoft.com... > > > Hi Sam, > > > > > > Thanks for your answer. There are few points I would like to point out > > > regarding your answer: > > > > > > Generally, I intend to update all the properties of any object. However, I > > > don't want to update (if I don't have to) two types of properties: > > > > > > Collections -usually contains other objects. I would like to > > > insert/update/delete only the objects in the collection that were modified > > by > > > the client. Some of my objects contain rather large collections and some > > may > > > contain more than one property which is a collection. > > > > > > Large sub-objects - that is an object which is contained in the main > > object > > > and is large enough to reconsider updating it for no reason. > > > > > > Performance is indeed a consideration but another motive was the DB > > > concurrency. Naturally those sub-objects and collections usually reside in > > > other tables in the DB and may have been modified but other users. > > Although I > > > have a mechanism to prevent it, I've learned that those big updates which > > > wipe everything an re-insert new date usually turn to be messy. > > > > > > Although the "dirty" flag is a possibility, it is a "dirty" one. :-) And > > in > > > case of external systems/other UI that will update my system, I will have > > to > > > make sure that the flags were used. > > > > > > I was thinking about having my business component in the middle tier > > > comparing the cached object with the modified object. According to the > > > comparison results the relevant main-object and/or sub-objects and/or > > > collections objects will be sent to the data layer components. The > > > collections can be serialized to XML strings and sent to the DB as such. > > But > > > this seems like a complex solution to what I imagined should be quite
Hello,
Regarding the issue, I would like recommend you use DataSet instead. In a Dataset, we can easily find if a row or a table has been changed or deleted. And you don't need to keep a copy in middle layer and compare the arraylist. DataSet may bring a distinct improvement on the performance. To use dataset, you may need to change your design, use Dataset instead of arraylist, and pass dataset to clients and get them back for update. Luke
Hi Luke,
Thanks. Would you still recommend datasets when remoting is in use? My client connect with the middle tier using remoting and as far as I know dataset tend to be quite havy for passing it on the wire. Actually, I was even considering converting the ArrayList to plain arrays before passing them up or down. But maybe the performance of the comparison is actually worse then passing dataset? Thanks, Gwenda [quoted text, click to view] "[MSFT]" wrote:
> Hello, > > Regarding the issue, I would like recommend you use DataSet instead. In a > Dataset, we can easily find if a row or a table has been changed or > deleted. And you don't need to keep a copy in middle layer and compare the > arraylist. DataSet may bring a distinct improvement on the performance. > > To use dataset, you may need to change your design, use Dataset instead of > arraylist, and pass dataset to clients and get them back for update. > > Luke >
Hello,
DataSet serializes as XML,Passing large DataSets over a remoting channel can consume large amounts of processor and network resources. From this point, an array is better than dataset. But if you need conversion on array, it is hard to tell which is better on performance. You may need to test these two methods with your actual data for sure. Generally, I prefer to the dataset solution since it need less work, for example, we don't need to keep a copy and compare row by row. Here is some useful aticle you may refer to: Designing Data Tier Components and Passing Data Through Tiers http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/ BOAGag.asp Improving Remoting Performance http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag/html/ scalenetchapt11.asp Luke (This posting is provided "AS IS", with no warranties, and confers no rights.)
Hi Luke,
Thanks again. I'll probebly go for the dataset - being simpler solution as you said. But i will give it a test to see how it goes with the remoting. Gwenda [quoted text, click to view] "[MSFT]" wrote:
> Hello, > > DataSet serializes as XML,Passing large DataSets over a remoting channel > can consume large amounts of processor and network resources. From this > point, an array is better than dataset. But if you need conversion on > array, it is hard to tell which is better on performance. You may need to > test these two methods with your actual data for sure. Generally, I prefer > to the dataset solution since it need less work, for example, we don't need > to keep a copy and compare row by row. > > Here is some useful aticle you may refer to: > > Designing Data Tier Components and Passing Data Through Tiers > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/ > BOAGag.asp > > Improving Remoting Performance > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag/html/ > scalenetchapt11.asp > > Luke > > (This posting is provided "AS IS", with no warranties, and confers no > rights.) >
Hi Gwenda,
Thank you for the reply. Good Luck! Luke
Don't see what you're looking for? Try a search.
|
June 2003
July 2003
August 2003
September 2003
October 2003
November 2003
December 2003
January 2004
February 2004
March 2004
April 2004
May 2004
June 2004
July 2004
August 2004
September 2004
October 2004
November 2004
December 2004
January 2005
February 2005
March 2005
April 2005
May 2005
June 2005
July 2005
August 2005
September 2005
October 2005
November 2005
December 2005
January 2006
February 2006
March 2006
April 2006
May 2006
June 2006
July 2006
August 2006
September 2006
October 2006
November 2006
December 2006
January 2007
February 2007
March 2007
April 2007
May 2007
June 2007
July 2007
August 2007
September 2007
October 2007
November 2007
December 2007
January 2008
February 2008
March 2008
April 2008
May 2008
June 2008
| home · blog · groups | Questions? Comments? Contact the dn |