all groups > sql server programming > august 2005 >
You're in the

sql server programming

group:

Handling large amounts of data for reporting?


Re: Handling large amounts of data for reporting? AK
8/26/2005 2:37:47 PM
sql server programming: Do you need to report against a big table that grows 100K rows daily or
do you need to report against a table that has 100K rows?

100,000 rows of data is usually NOT a large amount on decent hardware.
Re: Handling large amounts of data for reporting? Aaron Bertrand [SQL Server MVP]
8/26/2005 5:33:13 PM
[quoted text, click to view]

Usually, you can run queries in the background that will aggregate to the
various dimensions you want to measure (e.g. ).

In one current system, we aggregate data by hour, then when someone wants to
do a "this week" or "this month" type of query, we simply union the
aggregated data (super fast) from before the last hour where stats were
rolled up, against the raw data (which is also relatively fast, using the
same kind of query that would "roll it up") and then apply group by to the
derived table. This way the majority of the crunching is already done, and
you are applying operations on raw data to only the last hour or less.

You might also look at Analysis Services or other BI products, but I think
the lag of rolling data into cubes/dimensions will fail your realtime
requirement.

Re: Handling large amounts of data for reporting? Stu
8/26/2005 6:33:19 PM
You may also want to explore partitioned views depending on the types
of queries you're looking at; we partion our data daily (about 800k
rows a day), and the optimizer knows wehre to look for the data we want
to retrieve.

Stu
Handling large amounts of data for reporting? Lucas Tam
8/26/2005 9:07:23 PM
Hi all,


I have an application which logs a considerable amount of data. Each day,
we log about 50,000 to 100,000 rows of data.

We like to report on this data... currently I'm using a stored procedure to
calculate the statistics, however since this is an ad hoc, reports take a
while to generate.

So how do you guys handle large amounts of data? Is there a good way to
precalculate a a set of statistics to handle ad hoc queries (i.e. By Hour,
By Day, By Week, By Month). Our application also provides near realtime
statistics... so precalculation has to be done on a continual basis. Does
..NET have any statistics classes that might help out with this sort of
thing? I don't think .NET performance counters will work since they don't
log persistent data.

Any ideas?

Thanks!

--
Lucas Tam (REMOVEnntp@rogers.com)
Please delete "REMOVE" from the e-mail address when replying.
Re: Handling large amounts of data for reporting? Lucas Tam
8/26/2005 9:56:04 PM
"AK" <AK_TIREDOFSPAM@hotmail.COM> wrote in news:1125092267.248510.101290
@g49g2000cwa.googlegroups.com:

[quoted text, click to view]

I need to report against a table that grows by ~100K row daily.


--
Lucas Tam (REMOVEnntp@rogers.com)
Please delete "REMOVE" from the e-mail address when replying.
Re: Handling large amounts of data for reporting? JT
8/29/2005 10:30:49 AM
Summarized datasets, federated views, and OLAP cubes some of the basic
components, but the general solution is called data warehousing. Basically,
the concept is to export a pre-calculated and pre-summarized subset of data
in the OLTP system another database or server for the purpose of reporting
or analysis. The goal is to design a system that is optimized to suit your
own specific analytic needs, so the appropriate implementation details can
very from one warehouse to the next.
This document provides a good overview:
http://userfs.cec.wustl.edu/~cse530/2004/Data-Warehousing-Combined.ppt


[quoted text, click to view]

Re: Handling large amounts of data for reporting? Lucas Tam
9/14/2005 8:55:32 AM
"JT" <someone@microsoft.com> wrote in
news:Oq9egaKrFHA.3452@TK2MSFTNGP14.phx.gbl:

[quoted text, click to view]

Hmmm... that's the problem.

Alot of the data our application generates is being used by external
clients, and they all have differing reporting requirements.

Not to mention, management seems to change the way queries are calculated
every so often.

Any ideas?

Would the Star/Snowflake structures be my best solution?

Thanks!

--
Lucas Tam (REMOVEnntp@rogers.com)
Please delete "REMOVE" from the e-mail address when replying.

Got something to say? Click here to post a reply to this thread.
Don't see what you're looking for? Try a search.
View other messages in the sql server programming group.
AddThis Social Bookmark Button
View Other Months
January 2003
March 2003
April 2003
May 2003
June 2003
July 2003
August 2003
September 2003
October 2003
November 2003
December 2003
January 2004
February 2004
March 2004
April 2004
May 2004
June 2004
July 2004
August 2004
September 2004
October 2004
November 2004
December 2004
January 2005
February 2005
March 2005
April 2005
May 2005
June 2005
July 2005
August 2005
September 2005
October 2005
November 2005
December 2005
January 2006
February 2006
March 2006
April 2006
May 2006
June 2006
July 2006
August 2006
September 2006
October 2006
November 2006
December 2006
January 2007
February 2007
March 2007
April 2007
May 2007
June 2007
July 2007
August 2007
September 2007
October 2007
November 2007
April 2008
August 2008
home · blog · groups Questions? Comments? Contact the dn