Project Management

Testing the Performance and Scalability of Your Data Warehouse

Let me recap a familiar situation (one that I've been through as well!):

 

You've built a data warehouse for the first time and have officially launched to your user community of 50 users. Initially, query performance is slow, but you've added the indexes and tuned the queries and everyone is happy. You move on to the next iteration--adding more data, adding more functionality and adding more users. Slowly but surely, that performance begins to degrade once again.

 

There are no more queries to tune, no more indexes to add and no more money for hardware. This issue is happening at warehouses all across the country. The reason that you were unprepared for this situation is because you did not perform adequate performance testing and modeling once you had gone through that initial query tuning.

 

Tuning the warehouse for a single user/single query scenario and assessing performance for the multi-user/multi-query scenario is very different. In one case, you are looking at an EXPLAIN of an SQL statement and adding appropriate indexes or re-writing a query to make it return more quickly.

 

In the other case, you are examining what happens when many users access the system at the same time, executing multiple queries simultaneously. In the first scenario, you could be thrilled about the 10-second response time, but in your second scenario, that same …


Please log in or sign up below to read the rest of the article.

ADVERTISEMENT

Continue reading...

Log In
OR
Sign Up
ADVERTISEMENTS

"It's kind of fun to do the impossible."

- Walt Disney