What is Lean BI and its Purpose?
By: Steve Dine
I have yet to meet a BI team that has too little on its plate. In this article we will look at some of the drivers creating all this work and offer a set of principles that can be used to become more efficient and effective while still focused on delivering value. The main idea for this methodology evolved while leading a BI team for a leading medical manufacturer. We were frustrated by the amount of time spent maintaining our existing infrastructure and the number of features/data elements being added to the data warehouse that weren’t being utilized. During that time, the company was advocating Lean manufacturing principles and practices in our European divisions and there appeared to be some parallels to our situation. Our dabbling into Lean was the original root that grew into this concept.
Why is there so much pressure on BI programs?
When calculating the ROI of BI programs and BI projects we tend to underestimate resources. BI teams aren’t generally built, they evolve over iterations. We learned a number of years ago that the big bang approach to data integration simply doesn’t work. However, the alternative is that our programs slowly grow over time. We add more reports, more users, more ETL jobs, more data, more applications, more hardware and more overhead. Iterative development also causes architectural challenges. These include, but are not limited to:
- Non-scalable systems
- Rigid architectures
- Duplicate metadata
- Disparate scheduling
- Unleveraged metadata
- Additional & dis-integrated applications
- Multiple security architectures
- ‘Plug and play’ stovepipe solutions
- Non-existent data/systems governance
Yet, organizations often aren’t receptive to adding more resources to the BI team because the program was sold on reducing the number of resources and time required for reporting and analysis. The general solution is to push the work out to the functional areas of the organization by recruiting subject matter experts and data stewards. While this helps with governance, it rarely reduces the amount of pressure on BI departments.
Another challenge is that integration and reporting solutions generally take longer than users are willing to wait. Vendors recognize this and offer ‘point’, ‘packaged’ or ‘pre-modeled’ solutions that are sold directly to functional areas. These often include the hosted (Saas/ASP) solutions that are marketed as fast, simple and easy-to-use. These solutions are very tempting because they offer the opportunity to deliver reporting and analysis in significantly less time than BI groups can generally promise. However, they usually only target a specific subject area of data and over time, the functional areas inevitably require integration with other types of data. This causes more work for BI programs as they become entangled in supporting these solutions.
In many organizations, users require access to both strategic and operational (lower latency data). Heterogeneous data access and EII solutions are gaining popularity and adding complexity to our BI environments. Also, companies are utilizing hosted solutions for operational needs and purchasing large amounts of external data for use in their organization. In many cases, the companies supplying external data also offer their own reporting solutions and refuse to make their data available to BI programs. This can be a challenge to maintain as users begin to require additional functionality and integration with existing BI applications.
What is Lean BI?
Before we delve into what Lean BI is, it is important to address what Lean BI is not. Many people hear the word ‘Lean’ and it conjures up images of featureless tools, limited budgets, reduced development and the elimination of jobs. Dispelling those myths out of the gate is crucial in order to garner support for implementing Lean BI from the organization and the BI team. If team members feel that by becoming lean they are working themselves out of a job then they will not support your efforts. If your customers feel that they will receive less service or be relegated to using suboptimal tools then they may not support your efforts as well.
So, what is Lean BI? Lean BI is about generating additional value by accomplishing more with existing resources by eliminating waste. Lean BI is a set of principles and practices that have been influenced by 3 main concepts:
- Lean Manufacturing
- Systems Theory
- Agile Project Management
Lean Manufacturing is a set of principles and practices which evolved from the Toyota Production System. It has helped companies, like Toyota, become one of the top car manufacturers in the world. It focuses on identifying customer value and then delivering more with existing resource by eliminating waste in the organization. Waste is defined as any human activity which absorbs resources but creates no value. (Jones) By freeing up time usually consumed by wasted effort organizations can focus on spending more time creating customer value. While not all Lean Manufacturing concepts apply to BI and not all Lean BI concepts apply to manufacturing. While some of the principles are more closely aligned with the shop floor, others are universal to all functional areas.
Waste in BI programs is defined as any activity, task, process, mapping, object, code, report or data which absorbs resources but creates no incremental value to the customer. In my experience, BI programs are affected by both structural forms of waste and local forms of waste. Structural forms of waste exist in organizations that are more difficult to change such as compliance activities, processes applied universally to very different activities, vendor impositions, and hierarchical control. These create additional work for the BI team that often produces little value to the customer. They include waiting for management signoff, waiting for external QA resource to become available and maintaining little used functional reporting applications. Local forms of waste are activities that we engage in that are within the BI team’s control to change without engaging other areas of the organization. Examples of local waste include maintaining the same business rules & metadata elements in more than one application, ETL process steps that are no longer required and avoidable mistakes which require rework.
System Theory focuses on the theory, methods, and philosophy needed to analyze the behavior of systems in management and other fields. (Systems Thinking and the Lessons of the Last 35 Years) It recognizes that systems, such as BI programs, are complex and that we must consider this fact when developing our architecture and making decisions. A system consists of interdependent and interacting parts joined by a purpose (Poppendieck).
The ability of a system to achieve its purpose depends on how well the parts work together, not just how they work individually. Systems theory also recognizes that the decisions we make and actions we take affect other parts of the system that we don’t necessarily consider. Decisions made in other parts of the organization often affect BI teams, such as deciding to contract with an outside vendor to host an application that doesn’t allow access to their data. BI programs grow through iterations and so we need to consider how our architecture will change over time so can augment versus replace.
Agile Project Management evolved from the Agile Manifesto for Software Development. Much has been written about Agile, including myself, and its application to BI programs and projects (see ‘Who doesn’t want to be Agile?’ http://www.b-eye-network.com/channels/1536/view/8171/). Agile was originally designed for software development but many of its concepts can be applied to BI projects. It is important for BI teams to be agile because businesses are not static and we must be able to effectively deliver value in a changing environment. Effectively delivering value means reducing the time between request and rollout and only delivering what is required. The challenge with the traditional waterfall approach is that between the time of requirements analysis and delivery, many of the requirements are no longer required because the business has changed. This is a great example of waste in BI programs.
Delivering more value with existing resources is at the core of Lean BI. As the economy tightens, BI programs will need to adopt new principles and practices if they are to flourish. In future articles, I will reveal the 6 principles of Lean BI and discuss the real-world practices, from the trenches, that can be employed to reduce waste in any BI program or project.
Jones, W. a. (n.d.). Lean Thinking.
Poppendieck. (n.d.). Lean Software Development.
Systems Thinking and the Lessons of the Last 35 Years. (n.d.). Forrester.
Are you ready to discuss your project?
Let's chat about it, we look forward to helping and becoming your data partner.