Optimize large INSERT -- sql server dts

sql server dts : Optimize large INSERT

Dave
2/14/2006 2:19:22 PM

Can DTS outperform pure T-SQL on inserts when the source and
destination are in the same database?

I have a staging table with the same schema as my destination table.

My destination has 200 million records and my staging table has 30
million records.

Do you have any suggestions on indexing strategies or optimization
tips?

Currently the clustered index is a covering index on (col2_int,
col3_int, col1_char_55)

SELECT count(*)
FROM myTable_stage a with(nolock)
LEFT OUTER JOIN myTable b with(nolock) ON b.col1_int = a.col1_int
AND b.col2_int = a. col2_int
AND b.col3_char_55 = a.col3_char_55
WHERE b.col1_int IS NULL

col1_int has 1500 unique values
col2_int has 50000 unique values
col3_char_55 has 50000 unique values

Currently I am using one copy column transformation in 50000 batches.

Dave
2/15/2006 8:59:29 AM

No the problem is inserting 30 million records in general and
understanding how the DTS copy column transformation using the fast
load option compares to native the T-SQL insert.

Books online claims that DTS uses a buffer when it commits the
transformation in batches. However it is taking so long to do the
insert, I wonder if the transformation really writes 30 million records
to a buffer or if it re-queries the data after 1 or more batches.

I was only interested in the best index strategy for the insert step so
I could test and have a baseline.

Also, why can't clustered indexes be covering indexes? This makes no
sense to me at all.

Evergray
2/15/2006 5:19:08 PM

The problem is 50K inserts? Then the way may be Source --> transform -->
temporary table --> insert (dest) select * from temp. table

What transforms and inserts are performed? Speaking abount indexing strategy
without knowlege about queries is nonsense.
Also, clustering index can't cover queries by definition.

--
WBR, Evergray
--
Words mean nothing...

[quoted text, click to view]

Got something to say? Click here to post a reply to this thread.

Don't see what you're looking for? Try a search.

View other messages about sql server dts

View Other Months
June 2003
July 2003
August 2003
September 2003
October 2003
November 2003
December 2003
January 2004
February 2004
March 2004
April 2004
May 2004
June 2004
July 2004
August 2004
September 2004
October 2004
November 2004
December 2004
January 2005
February 2005
March 2005
April 2005
May 2005
June 2005
July 2005
August 2005
September 2005
October 2005
November 2005
December 2005
January 2006
February 2006
March 2006
April 2006
May 2006
June 2006
July 2006
August 2006
September 2006
October 2006
November 2006
December 2006
January 2007
February 2007
March 2007
April 2007
May 2007
June 2007
July 2007
August 2007
September 2007
October 2007
November 2007

home · blog · groups

Questions? Comments? Contact the dn