For over 15 years, I have been working with small to very large-scale data warehouses and have become quite comfortable with their vernacular and approaches for design, development, and implementation. Over the last few years there has been a shift in the marketplace around the traditional concept of data warehousing. Many organizations’ data warehouses have served them for years, but with general maintenance and the inevitable changes in business, companies are now ready to replace their current technology with an architecture that will meet both current and future data needs. Many of the initial conversations around developing a new, upgraded data warehouse are scoffed at and immediately dismissed. The term “data warehouse” has been somewhat relegated to the “old school” bucket of terms next to “client server”, “host computing” and “structured programming”. Organizations are looking for new, cutting edge data platforms.
Many consultants have come across unexpected data conditions while deep into the development or testing phase of a data integration effort. Unfortunately, the need for a major shift in architecture or approach is sometimes identified soon after a system is implemented into production. Upon discovering this type of situation, you likely experience a sinking feeling, knowing the potential negative consequences. You imagine blown budgets, missed deadlines, loss of credibility with the business, and unpredictable results. You begin analyzing what caused the problem, how it could have been avoided, and who is at fault
This is the third in a series of articles describing foundational steps to enable agile data warehouse development. This first series of articles describe foundational steps that enable agile data warehouse development – something that has been a challenge in enterprise data management for years. My prior articles published thus far describe how to develop a Business Conceptual Model as a starting point, then building a “grass roots” (at a minimum!) Data Governance capability.
The next focus for setting yourself up for a best in class agile data warehouse environment is to develop a high level data flow architecture that is inherently flexible and leverages repeatable design patterns.