Business Intelligence Review of 2013
Each year, we reflect upon the business intelligence (BI) and data integration (DI) industry and provide a Business Intelligence Review of the noteworthy trends that we encounter in the trenches. Our Business Intelligence Review emanates from five sources: our customers, industry conferences, articles, social media, and BI software vendors. This years Business Intelligence Review has proved to be an interesting one, on many fronts. Here are our observations for 2012 and our expectations about 2013.
1. Business Intelligence Review - Programs Matured
As we reported in 2011, larger numbers of existing BI programs are continuing to mature beyond managed reports, ad hoc queries, dashboards, and OLAP. Companies are increasingly looking to derive more value from their data via technologies and capabilities such as:
- Text and social analytics
- Advanced data visualization
- Predictive & descriptive analytics
- Geospatial analysis
These capabilities all require significant computing power, and consequently we have seen a corresponding rise in technologies such as analytic databases and analytic applications. We see this trend continuing into 2013 and beyond.
2. Business Intelligence Review - Greater focus was placed on operational BI
Over the past few years, we have observed an increased focus on operational BI. As corporate BI programs mature, this is a natural evolution. There is considerable value in using BI to support and enhance operations within a company, but companies that are successful doing this must realize that operational BI is a different class of data. Operational BI generally requires lower data latency, higher data selectivity, and a larger amount of query concurrency than traditional analytic workloads. These factors often require a different architecture than what was designed for the data warehouse.
Furthermore, support of the system may need to be executed differently. If a load on a traditional data warehouse fails, it is often acceptable to address it within hours, not minutes. For operational BI, a 24/7 support model is more often required because load failures may immediately impact the bottom line.
We see the trend toward operational BI and lower latency analytics continuing as organizations broaden their focus from enterprise data warehousing to enterprise data management.
3. Business Intelligence Review - BI wanted to be agile
We've always recognized the high cost of, and long lead times for, implementing BI, but customers have finally said “enough” and BI teams have to listen. New software-as-a service (SaaS) BI offerings and departmental solutions enable businesses to move forward without IT, putting even more pressure on BI programs to deliver results faster. Businesses are looking for new ways to implement BI and are finding that many agile practices (smaller, focused iterations; daily scrum meetings; embedded business representatives; prototyping; and integrated testing) help accelerate BI projects and bolster communication between business users and IT. Certain technologies are also helping influence this shift. Data virtualization, for example, allows a “prototype, then build” capability and doesn’t require physicalizing all the data required for analysis. However, agile was created for software development, not BI, and early adopters are learning that there are many differences. For example, the tools to automate software code testing are far more numerous and mature than for ETL mappings and data warehouses.
We expect to see BI practitioners continue to refine which agile principles are effective with BI and which ones don't translate as well. We also expect to see a rise in enabling technologies, such as desktop analytic software, data virtualization, and automated testing/data validation.
4. Business Intelligence Review - Momentum shifted to “in-memory”
With the rise of 64-bit architectures and the ever-decreasing cost of memory, we have recognized a shift in momentum from in-memory applications to the database (namely with data warehouse appliances). The past several years have seen the rapid ascent of in-memory analytic technologies. Tools such as Tableau, IBM TM1, Oracle TimesTen, Spotfire, and Qlikview have offered the promise of near-instantaneous analytic response.
However, as memory size continually increases and the cost of memory decreases, databases such as SAP HANA, Oracle Exalytics, Netezza, Kognitio and Teradata have been able to optimize their platforms to maintain more data in memory. This has resulted in significantly faster database operations for data residing in memory while allowing larger data sets to still reside on disk. This hybrid capability allows administrators to tune their environments to their specific analytic workloads.
Although we see the trend toward in-memory databases increasing, we also recognize that data volumes are increasing faster than the cost of memory decreases. We don’t foresee all enterprise data being stored in memory anytime soon; therefore, we believe there will soon be a greater focus on temperature-based storage management solutions in 2013.
5. Business Intelligence Review - Failed BI projects remained a challenge
Although the industry has learned and documented the reasons projects fail, it hasn't done much to stem the tide of failed projects. From our perspective, the overarching reason is that implementing successful BI projects is difficult, requiring a balance of strong business involvement, thorough data analysis, scalable systems and data architectures, comprehensive program and data governance, high-quality data, established standards and processes, excellent communication, and BI-focused project management.
From our perspective, we don't necessarily see this trend changing unless project teams:
- Institute and enforce enterprise data management practices.
- Ensure high levels of business involvement for BI projects.
- Institute measurable, value-driven metrics for each BI project.
- Change their mindset from offshoring BI projects to “smart-sourcing” them. It is important to not treat outsourcing as an all-or-nothing concept, but rather intelligently outsource components that work well (e.g., simple staging ETL, simple well-defined reports, operational support).
6. Business Intelligence Review - BI continued moving to the cloud
Cloud BI has continued to evolve and expand over the past several years, but more slowly than expected. A recent Saugatuck Technology survey concludes that only about 13 percent of enterprises worldwide are harnessing cloud-based BI solutions. However, vendors are innovating technology to address traditional deficiencies and hindrances such as data security, a lack of meaningful ETL capabilities, and performance challenges (of both hardware and the network). They are also addressing concerns that “infrastructure-as-a-service” may be too complex for many BI programs to consider.
To address these concerns, we have observed the emergence of an increasing number of vendors in the SaaS space. Solutions such as MicroStrategy Cloud, Microsoft Azure, Informatica Cloud, Pervasive Cloud Integration, and GoodData have begun to address these difficulties. These products show great promise, and over the coming years we expect them to continue to mature and validate the technical and fiscal viability of the space.
A note of temperance: a key challenge that still looms is the ability to effectively and efficiently integrate these solutions with existing security infrastructure (such as LDAP and AD). Until then, an additional security layer will be required to support BI in the cloud.
7. Business Intelligence Review - Interest in big data/Hadoop grew
Everyone has heard a lot about big data and Hadoop. This is understandable because data volumes continue to grow and technologies must innovate to support the increasing volumes in a cost-effective way. Furthermore, these technologies have been very well marketed; vendors have invested significantly in creating buzz.
However, like most emerging technologies, the capabilities are still not well understood by many BI organizations. The technologies do certain things very well but are typically not a wholesale replacement platform for a traditional data warehouse. Over the coming years, we expect companies to better understand how these technologies fit and leverage them accordingly.
Business Intelligence Review - Final Thoughts
This past year was certainly a year of change in the industry. We realize that a broad range of technologies, trends, and capabilities emerged in 2012. Our industry review is based on what we observed in the trenches. This doesn't always align with what industry analysts and vendors chose to promote. Rather, our observations tend to be more practical and focus on trends and technologies that are currently making a difference on the ground. We are curious to know if you agree. Please contact us and let us know what you think at firstname.lastname@example.org.
For previous copies and articles of the Business Intelligence Review, see our blog or conduct a search on our website. The Business Intelligence Review is published the first quarter of every calendar year.
Are you ready to discuss your project?
Let's chat about it, we look forward to helping and becoming your data partner.