But is that really true? In my experience, the only thing holding back a data warehouse from being able to provide near realtime data was not the data warehouse itself, but all the heavy software development processes around it, and a lack of communication with the business about the efforts put in to maintain good data quality. Both of these things are correctable, not just to provide near realtime BI, but to provide better BI in general.

Heavy Software Development Processes

This is almost always the result of doing software development using the waterfall approach. There are better ways (see: Enough Projects Drowned Under the Waterfall). By employing a software development methodology that doesn’t bury itself in hand-crafted documentation (which always requires a lot of time), and takes a multi-streaming approach to getting things done (like implementing Data Governance or taking advantage of it if it already exists), the development cycle can be shortened and more reactive to change. It also helps to have this coupled to an architecture that can support change and extension easily.

Communication with the Business

Often, in an attempt to get around the lack of communication with the development group, business users will build their own solutions to generate the BI they need quicker. This, of course, works because they are able to side-step all the (historically) heavy processes the data warehouse team has in place. What happens though, is that the solutions they build don’t take into account an enterprise view of the data, and always end up duplicating data transformation logic that already is or at least should be done somewhere else. This results from a lack of communication between the groups, and the lack of understanding between the business and the data warehouse team about the what is really needed and how what his really needed can be produced.

We Need to Talk and We Need to Act, at the Same Time

As mentioned above, the way out is to talk together, often and purposefully, and to act together, often and purposefully. But don’t wait for the talking to be done before you act, and don’t wait for the actions to be completed before talking again. Multi-streaming these activities makes them both go faster and results in a better solution through improved collaboration. Now that’s Agile Development!

Isn’t it time to restrict the obedience to the business side deciding about what is being developed? Shouldn’t IT considerations at least be as seriously considered when it comes to how to deliver the right systems and quickly? Agile methods are particularly appropriate for BI systems which are close to the end users. That users often cannot precisely formulate their needs initially, is one of the strongest arguments for an incremental and iterative process. But there are aspects specific to BI favouring prototyping. Encouraging active user involvement throughout the development is certainly of high value for both sides. Delivering the parts of highest priority first and using that output can give very useful input for subsequent phases including the option of stopping the project, if first results made the project obsolete or changed the requirements drastically. Among the strongest critisms BI teams face are being slow and not delivering very useful results. Both issues have been strong motivations for agile approaches since the beginning.

As Wikipedia puts it, agile development is “based on iterative development, where requirements and solutions evolve through collaboration between self-organizing cross-functional teams”. It is in particular “a business approach that aligns development with customer needs and company goals.”

We might need to reinterpret some of the principles of the agile manifesto. The principle working software is delivered frequently might sometimes be read results are delivered frequently. It is the close collaboration between the BI team and the users supported by frequent output to the users keeping them up to date that matters here. A principle that seems very hard to swallow and contrary to most IT development culture says that even late changes in requirements are welcomed. Ok, maybe not all changes are appropriate and need to be expressed so late. But this openess and flexibility is key to agile development. It means accepting that it is hard to impossible for users to sign and close BI requirements before having seen the data.

The fact that relatively few organisations apply agile processes probably does not mean that people are all happy with the waterfall process. Many large organizations have difficulty bridging the gap between the traditional waterfall method and an agile one. It would also require getting around reasons predominantely external to BI such as overall budgeting, planning and roadmap management and prioritisation.

Wikipedia describes the agile home ground as projects of low criticality with a small team of senior developers and requirements that change very often. A consultative management culture would help.

Don’t get me wrong. Major infrastructure projects, be it enterprise data warehouse or something operational, are not good initial candidates for process innovation. Where the big picture is important to be kept in mind, where the long term maintenance and flexibility has to be assured and users interact with the deliverable only indirectly, a classical planned approach still has its virtue. But when it comes to small analytical marts and BI interfaces we should take a good dose of self-confidence and go agile even it takes a few serious discussions.