For over 15 years, I have been working with small to very large-scale data warehouses and have become quite comfortable with their vernacular and approaches for design, development, and implementation. Over the last few years there has been a shift in the marketplace around the traditional concept of data warehousing. Many organizations’ data warehouses have served them for years, but with general maintenance and the inevitable changes in business, companies are now ready to replace their current technology with an architecture that will meet both current and future data needs. Many of the initial conversations around developing a new, upgraded data warehouse are scoffed at and immediately dismissed. The term “data warehouse” has been somewhat relegated to the “old school” bucket of terms next to “client server”, “host computing” and “structured programming”. Organizations are looking for new, cutting edge data platforms.
Many consultants have come across unexpected data conditions while deep into the development or testing phase of a data integration effort. Unfortunately, the need for a major shift in architecture or approach is sometimes identified soon after a system is implemented into production. Upon discovering this type of situation, you likely experience a sinking feeling, knowing the potential negative consequences. You imagine blown budgets, missed deadlines, loss of credibility with the business, and unpredictable results. You begin analyzing what caused the problem, how it could have been avoided, and who is at fault