In a recent article by Wayne W. Eckerson, Director, TDWI Research, entitled “Operational BI: Sorting Out Your Options”, the options are given for providing near realtime business intelligence. The point of the article seems to be that you can have your BI either fast or of high quality, but not both. Given that a data warehouse by definition is not a near realtime animal, and providing near realtime data means excluding data quality checks, data integration checks and all the other things that add latency to the data warehouse, it seems that you can’t have both.
But is that really true? In my experience, the only thing holding back a data warehouse from being able to provide near realtime data was not the data warehouse itself, but all the heavy software development processes around it, and a lack of communication with the business about the efforts put in to maintain good data quality. Both of these things are correctable, not just to provide near realtime BI, but to provide better BI in general.
Heavy Software Development Processes
This is almost always the result of doing software development using the waterfall approach. There are better ways (see: Enough Projects Drowned Under the Waterfall). By employing a software development methodology that doesn’t bury itself in hand-crafted documentation (which always requires a lot of time), and takes a multi-streaming approach to getting things done (like implementing Data Governance or taking advantage of it if it already exists), the development cycle can be shortened and more reactive to change. It also helps to have this coupled to an architecture that can support change and extension easily.
Communication with the Business
Often, in an attempt to get around the lack of communication with the development group, business users will build their own solutions to generate the BI they need quicker. This, of course, works because they are able to side-step all the (historically) heavy processes the data warehouse team has in place. What happens though, is that the solutions they build don’t take into account an enterprise view of the data, and always end up duplicating data transformation logic that already is or at least should be done somewhere else. This results from a lack of communication between the groups, and the lack of understanding between the business and the data warehouse team about the what is really needed and how what his really needed can be produced.
We Need to Talk and We Need to Act, at the Same Time
As mentioned above, the way out is to talk together, often and purposefully, and to act together, often and purposefully. But don’t wait for the talking to be done before you act, and don’t wait for the actions to be completed before talking again. Multi-streaming these activities makes them both go faster and results in a better solution through improved collaboration. Now that’s Agile Development!