all groups > sql server full text search > june 2004 >
You're in the

sql server full text search

group:

Incremental population works too slow


Incremental population works too slow VikS
6/14/2004 12:42:21 PM
sql server full text search:
Hi,
We have two SQL machine (Win2000+SP4, SQL2000+SP3).
One of them is main production server, the second one is
backup machine.
We have big table there (34000000)and full-text index
based on that table.
We change this table everyday.
Last week the changes were less than 30 records.
On both machine I made full-text rebuilding (with the
same priority 5)(There were not additional jobs on any
machines).
Complete rebuilding worked 12 hours.
Incremental population worked 24 hours. WHY? it is only
30 records.
Secondly, If we make incremental population in production-
Would full-text catalog accessible in that time?
And what happened if incremental population is still not
finished making incremental population and we make
changes again and again and again...?
Would full-text catalog workable? Or will it freze?

Re: Incremental population works too slow Hilary Cotter
6/14/2004 11:01:43 PM
An incremental population is very similar to a full population. In both of
them all rows are extracted. In a full population all rows are indexed, in
an incremental population all the rows are extrated and then MSSearch does
some analysis to find out which rows are changed, inserted or deleted. So if
a large number of rows are changed you are better off using a full
population.

Microsoft recommends you use change tracking to get around the poor
performance of both full and incremental indexing.

The catalogs will be available when doing both full and incremental
populations after the initial full population is run. You can also access
the catalog while the initial population is being done, but the results
aren't predictable.
--
Hilary Cotter
Looking for a book on SQL Server replication?
http://www.nwsu.com/0974973602.html


[quoted text, click to view]

Re: Incremental population works too slow Viktor Soubbotine
6/15/2004 5:58:39 AM


Thank you Hilary.
There is some small problem about allowing tracking on that table.
There is one more SQL server and the both these servers got
this big table with full-text indexing from this one through
transactional replication.
Daily update (as usual) is not more than 100 records.
But once a month an update could be 200000 rows.
And in that case we will get very big problem with tracking (I think).
Any ideas?

*** Sent via Devdex http://www.devdex.com ***
Re: Incremental population works too slow Hilary Cotter
6/15/2004 11:04:01 AM
Last time I tried change tracking on a large update performance was acceptable, this was with about 10% changes.

One thing to watch out for is locking on the table with the insert process and the indexing process. In cases like this you might want to select the update index in background option.

--
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html




[quoted text, click to view]
Re: Incremental population works too slow Phil Sherrod
6/15/2004 3:17:06 PM

[quoted text, click to view]

Excuse me for butting into this thread, but we are experiencing a similar
problem with the incremental indexing. In our case, our database currently
has 3.2 million rows, but we expect it to grow to about 6 million rows. We
are adding about 200,000 rows per day, and once we reach full size we will
add and delete 200+ k rows/day.

Currently, the "incremental" index build takes over 8 hours, and it is
getting slower each day. Do you think change tracking would be suitable for
our situation with 200+ k insertions and deletions per day?

Currently we have two 7200 RPM IDE disks in a RAID 0 array. Do you think a
caching RAID controller would significantly improve performance? How about
switching to SCSI?

Thank you for your help.

--
Phil Sherrod
(phil.sherrod 'at' sandh.com)
http://www.dtreg.com (decision tree modeling)
Re: Incremental population works too slow Hilary Cotter
6/16/2004 9:41:29 AM
It should work. 200,000 inserts per day works out to be about 417 inserts per minute if you distribute these updates over a 8 hour period.

I would advise you to test.

I did some testing on a IDE Raid controller - adaptec 1200 with 4 drives and placed my catalog on this. I believe I got a 10% increase in performance while indexing.

Placing my database on a Raid 5 drive (adaptec 2000s) I got a 23% increase in performance in conjunction with the IDE raid controller.
--
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html




[quoted text, click to view]
AddThis Social Bookmark Button