Slowly, more and more analyst firms and vendors draw a more differentiated picture and accept that there is a large range of user situations and needs which ask for a range of appropriate technical answers.

One of the most important goals of the data warehouse is to provide a single version of the information. In order to avoid potentially different data selections or transformations it would only be sensible to also have just one tool to access the single version of the information, eradicating the problem of different numbers for the same question once and for all. This might be the strongest motivation for BI professionals and business users who seek a universal BI end user tool that answers all business needs and widens the access bottle neck. The growing market opportunity for BI tools motivates the software vendors to promise technological breakthroughs and even miracles where required.

The challenge becomes apparent when taking a closer look at what different users need. At least three use cases need to be distinguished and give rise to different and conflicting software requirements:

• „Most data warehouses will probably form the basis for a stream of ready-made reports being send out to many users across the organisation. Usually no interactivity is provided with the reports and most users would more likely become confused by features allowing for customisation. What is needed are capabilities to automatically and repeatedly generate reports combining several tabular and graphical elements.
• „Senior management wants the convenience to oversee the entire business and drill down into subject areas when indicated. This need can be answered by information dashboards providing basic flexibility via an intuitive GUI. The bigger challenge here is to select and define the pieces of information to be displayed in line with the rest of the organisation.
• „Follow-up questions triggered by the two first user groups and all other analysis give rise to non-standard or ad hoc queries. They involve not only tools with full query flexibility but most importantly data analysis skills and knowledge of the data sources restricting these queries to a small minority of the users for most organisations.  Data warehouse thought leader Ralph Kimball wrote in 1998: “Ad hoc query tools, as powerful as they are, can only be effectively used and understood by about 10% of all the potential end users of a data warehouse. … The very best ROLAP-oriented ad hoc query tools improve the 10% number to perhaps 20%.” ^[1]
  Also data discovery tools, data mining workbenches and even plain SQL tools fall in this broad category.
  See also Stephen Few’s article Big BI is Stuck: Illustrated by SAP BusinessObjects Explorer (his Blog is highly recommended in general).

These three use cases could be further broken down, for example, by distinguishing between being OLAP focused or not, and vendors tend to showcase certain capabilities (such as data search) as constituting a whole new category.

It shows that a unified tool for all users and situations could probably only be developed at the expense of too many compromises and would probably satisfy no one in the end.

The even more utopian plan of BI delivery by pure self-service could only work with a complete data warehouse presented by a clearly documented user interface layer and in situations where queries are limited to simple look ups of stored facts. I’m not sure that the three conditions are fulfilled anywhere.

But is that really true? In my experience, the only thing holding back a data warehouse from being able to provide near realtime data was not the data warehouse itself, but all the heavy software development processes around it, and a lack of communication with the business about the efforts put in to maintain good data quality. Both of these things are correctable, not just to provide near realtime BI, but to provide better BI in general.

Heavy Software Development Processes

This is almost always the result of doing software development using the waterfall approach. There are better ways (see: Enough Projects Drowned Under the Waterfall). By employing a software development methodology that doesn’t bury itself in hand-crafted documentation (which always requires a lot of time), and takes a multi-streaming approach to getting things done (like implementing Data Governance or taking advantage of it if it already exists), the development cycle can be shortened and more reactive to change. It also helps to have this coupled to an architecture that can support change and extension easily.

Communication with the Business

Often, in an attempt to get around the lack of communication with the development group, business users will build their own solutions to generate the BI they need quicker. This, of course, works because they are able to side-step all the (historically) heavy processes the data warehouse team has in place. What happens though, is that the solutions they build don’t take into account an enterprise view of the data, and always end up duplicating data transformation logic that already is or at least should be done somewhere else. This results from a lack of communication between the groups, and the lack of understanding between the business and the data warehouse team about the what is really needed and how what his really needed can be produced.

We Need to Talk and We Need to Act, at the Same Time

As mentioned above, the way out is to talk together, often and purposefully, and to act together, often and purposefully. But don’t wait for the talking to be done before you act, and don’t wait for the actions to be completed before talking again. Multi-streaming these activities makes them both go faster and results in a better solution through improved collaboration. Now that’s Agile Development!

He also mentions, as I have in a previous article here, that master data management (MDM) systems can be used to remove from the traditional data warehouse the complex work of data integration. I see this happening more and more for the reasons I expressed earlier, mainly a reduced total cost of ownership and an increased usability of the information. MDM systems are perfectly suited to the “data integration” part of the overall business intelligence world, and as such, helps to allow the traditional data warehouse evolve to a smaller, quicker, and less expensive machine to operate. Also, sourcing a data warehouse from an MDM system means that analytical data can be provided far easier and quicker than before, and this supports Neil Radan’s BI 2.0.

I have worked with organizations that have come to the conclusion that the types, amounts and sources of information needed to run the business properly are constantly changing as the organization tries to keep up with or keep ahead of the competition in a more and more volatile business environment. What resulted was the organization stepping backward in the maturity of its data warehouse (according to the ” Data Warehouse Maturity Model” I mentioned in a previous article). One organization in particular, to get the data the business needs in front of them faster, did an end-run around the data warehouse and started pulling data from whatever source system they thought had what they needed. I have to admit that the web-based interface they built was very impressive, and they could indeed deliver information quickly.

The problem I had with it was simply that they had ignored all the “best practices” about doing good data integration, good data governance and good data management. Because they built their solution outside the normal project path of the organization, in order to avoid the lengthly project schedules and extra costs, they side-stepped everything the organization had put in place to mitigate just the kinds of problems that this new development brought up again. And the organization had suffered the same problems in the past! I was amazed that the business unit in question was allowed to do this, but in the end, you had to sympathize with them. All they wanted was the data in a form they could use, and quickly. The solution should have been to change the processes to allow for these kinds of tools, but still keeping the principles of proper data management in place.

So, they were trying to do Business Intelligence 2.0 by using Web 2.0 type features on top of an immature approach to enterprise data management. In the end, I think it will hurt them more than help them, but who knows. These days things change so fast inside and outside business’s that even problems like this can disappear amongst everything else that goes on.

I would like to know if anyone is or has worked somewhere where a proper implementation of “Business Intelligence 2.0″ is or  has happened. Please share your experiences in the comments section.

All of that reminded me of the Data Warehouse Maturity Model I have seen time and again at conferences I have attended and presented at. It shows that a data warehouse has reached its ultimate maturity when it achieves its potential for the return on the investment of building a data warehouse. That is, the business uses the data warehouse and the information within it to drive the business. I started to think about this while I was putting the business case together for this project. Too often, I have found that the decision makers in an organization are not always aware of the value of the data warehouse, and see it as a big, expensive, slow moving giant that they have to live with rather than really benefit from. It is beyond the scope of this article to discuss all the reasons why this might become the perception of the data warehouse, but it was sure the case here, which is why I needed to include in my business case the feedback from all the users of the data warehouse.

The organization in question was looking for ways to reduce the costs of the data warehouse and it occurred to me that one way to reduce the cost of it was to remove it all together from the business. I don’t mean throw it away (that would be insane), but why not get a little creative? Take all the hardware, the software, and the people that build, operate and maintain it all, and use it to spin off a separate business? This business would be initially funded by the parent organization, contracted by the parent to provide all the data warehouse services it did before but at a more favourable cost. It would operate as an independent business with the sole core competency of doing data warehousing, and be allowed to also sell its services to other organizations as well. By being wholly owned by the parent company and being able to expand its customer base beyond the parent, allows it to become a revenue centre for the parent. Eventually the separation would probably need to be more complete, I suppose, but still, it goes from being a cost centre to a revenue centre. Sweet!

Afterwards, I thought, hey, I’ve been doing software design and development for over 20 years, data warehousing for almost 10, and ran my own small consulting firm. Why don’t I just start up a business that provides a data warehouse as a service? Why, I thought to myself, that’s a great idea and would be a lot of fun! I know a couple of organizations that are moving in the direction of having other “systems as a service” anyway (at least outsourcing other major business support systems), so I could probable lock up a customer or two within a couple of years, and who knows? I started to seriously investigate the possibility. I would need to build/buy/rent a data center and build/rent/lease a bunch of servers and disk, not really a problem, but I would (ideally) need to come up with a data architecture that could support multiple clients, possibly across different industries. That would mean, design it, build it, test it, implement it, and then go sell it. Now that’s a lot of work. And I know that you can buy such industry specific data models. Teradata did, or maybe still does, have one, Oracle has one to, and I am sure there are others. Possible still, but a lot of work, and it would mean I might miss the opportunities I had because of the timing.

In continuing my investigation into setting all this up, I came across a company that already provides Data Warehousing as a Service (DaaS), called Kognitio. They have their own data centres, they have their own data models, and although they are the only ones I could find, but there might be others by now. So, someone had my genius idea before me (no surprise really). I still think that for an organization that is trying become more mature in their data warehousing, this is a good solution to get off the ground fast and jump the “gulf between ‘Inform Exectutives’ and ‘Empower Knowledge Workers’” in the afore-mentioned data warehouse maturity model. If you are looking to implement a new data warehouse, or re-architect one, or try to out-source you existing one, I suggest you give them a call. I think I might, just to see if they need an affiliate in Switzerland, where I live.

What do you thing of Data Warehouse as a Service? Good idea or bad? Easy or difficult to implement? Too big a change to your organization or not? Let us know your thoughts in a comment!

Isn’t it time to restrict the obedience to the business side deciding about what is being developed? Shouldn’t IT considerations at least be as seriously considered when it comes to how to deliver the right systems and quickly? Agile methods are particularly appropriate for BI systems which are close to the end users. That users often cannot precisely formulate their needs initially, is one of the strongest arguments for an incremental and iterative process. But there are aspects specific to BI favouring prototyping. Encouraging active user involvement throughout the development is certainly of high value for both sides. Delivering the parts of highest priority first and using that output can give very useful input for subsequent phases including the option of stopping the project, if first results made the project obsolete or changed the requirements drastically. Among the strongest critisms BI teams face are being slow and not delivering very useful results. Both issues have been strong motivations for agile approaches since the beginning.

As Wikipedia puts it, agile development is “based on iterative development, where requirements and solutions evolve through collaboration between self-organizing cross-functional teams”. It is in particular “a business approach that aligns development with customer needs and company goals.”

We might need to reinterpret some of the principles of the agile manifesto. The principle working software is delivered frequently might sometimes be read results are delivered frequently. It is the close collaboration between the BI team and the users supported by frequent output to the users keeping them up to date that matters here. A principle that seems very hard to swallow and contrary to most IT development culture says that even late changes in requirements are welcomed. Ok, maybe not all changes are appropriate and need to be expressed so late. But this openess and flexibility is key to agile development. It means accepting that it is hard to impossible for users to sign and close BI requirements before having seen the data.

The fact that relatively few organisations apply agile processes probably does not mean that people are all happy with the waterfall process. Many large organizations have difficulty bridging the gap between the traditional waterfall method and an agile one. It would also require getting around reasons predominantely external to BI such as overall budgeting, planning and roadmap management and prioritisation.

Wikipedia describes the agile home ground as projects of low criticality with a small team of senior developers and requirements that change very often. A consultative management culture would help.

Don’t get me wrong. Major infrastructure projects, be it enterprise data warehouse or something operational, are not good initial candidates for process innovation. Where the big picture is important to be kept in mind, where the long term maintenance and flexibility has to be assured and users interact with the deliverable only indirectly, a classical planned approach still has its virtue. But when it comes to small analytical marts and BI interfaces we should take a good dose of self-confidence and go agile even it takes a few serious discussions.

What I mean is that in the big picture of enterprise data management, there are data applications that produce data, and data applications that consume data. In a lot of cases, a data application can (and must) play both roles. I make the distinction here to show one of the great promises of master data management: the reduction in the amount of duplication of work being done by software processes that must transform data between the applications that produce that data, and the applications that consume it. In the following diagram, we can see what can happen if several data consumer applications (right side) must connect directly to other data producing applications (left side) to get (and transform) the data they need.

Not a Pretty Picture.

As you can see, it can get pretty messy, pretty quickly. There are things that exist in the IT world to try and reduce this complexity, SOA, EAI, etc. and some do a reasonable job, but in the end, there is more needed to really remove the duplication of effort. This is master data management (MDM). If we have an MDM hub in the middle of the above diagram, to act as a level of abstraction between data producers and data consumers, we can significantly simplify the picture and remove all redundancy.

Much Better.

The MDM hub works the most effectively, where there is an actual data store implemented within it. The efficiency arises from being able to separate source-system specific data transformations from source-system independent data transformations. It enables this by maintaining, at its core, a enterprise-level business entity view of the data held in the department level (purpose-specific) data applications. This business view of the data, is populated from the source system’s data using source-system specific transformations, and thereby transforming the data into a data model that represents how the business sees the data entities, rather than how the purpose-specific applications see the data. When the data is transformed in this way, into an enterprise level business view of the data, the enterprise as a whole can understand the data and use the data by doing further transformations of that data into purpose-specific data (i.e. data to improve marketing, customer care, logistics efficiency, etc.).  The beauty of the whole system is that the end-users of the data do not need to worry about directly transforming the source system data into their own purpose-specific data, especially when they might need the same type data from multiple source systems (each with their own way of storing that information). They only need to go to one place, where the structure of the data is oriented to how they see the data, and where the data means the same thing thing to everyone (enterprise level). They are then isolated from the changes underneath, that is if a source system is replaced, decommissioned, or radically changed, the impact is kept to a minimum. The only changes required are the source-system specific transformations, and nothing else.

To get back to my original argument, I believe that an MDM system is mandatory for efficient operation of any medium to large-sized organization, and gets more important as the enterprise gets bigger. And I like to keep the main roles of the data managers separate from one another (at least on a personnel level, it not at an organizational structure level). That is, I prefer to have those individuals who use the data to run the business (data consumer enterprise roles) focused on using the data, and those individuals who are responsible for producing the data, focused on the applications that produce the data, regardless of where they fit into the organization’s operational structure. Therefore, I maintain that those individual who are responsible for the MDM system focus on doing that to the best of their abilities, and those that use the MDM system, such as those who would use the MDM system as the main source for the data warehouse, focused on building and maintaining the data warehouse.

So, at the end of the day, the bulk (possibly) of your data integration would happen in the MDM system, using source-system specific transformations, which would then allow the data warehouse to focus on generating analytical data for downstream use , without having the added burden of being responsible for the data integration as well, having only to worry about using source-system independent transformations, like many other downstream applications.

The ultimate result of the data warehouse team having to do this reconciliation on an on-going basis, is the promotion of the idea of data governance at the enterprise level. Pushing the ownership of the meanings (thus the definitions of) the data within the data warehouse, back to the organization seems like the right thing to do, and of course, it is. It is, if it’s done properly, because what we really want to do by implementing some data governance in the organization, is to implement a master data management system, which data governance is an important part of. We are saying, “We know that the data must be managed at the enterprise level, not at the department level, and we must engage the entire enterprise in the management of the data.”.

The management of “data” at an enterprise level means removing the shackles of departmental ownership of the data while still maintaining the departmental systems that manage that data. This is, by far, the trickiest part, as it deals with people, rather than just the technology and we will examine this in a later post. For now, though, we need to understand the implications of what it means to “manage data at an enterprise level”. These implications include

• restructuring the organization a little to implement enterprise data governance,
• looking at, planning for, and possibly phasing in a master data management system,
• removing some or all of the responsibility for data integration from the data warehouse (controversial, I know),

to mention but a few. We will go into more details about these and other issues in future posts as well.

Becoming more mature as an enterprise in how we deal with our data requires a shift in the way we look at, work with, and manage the data that we need to run our enterprise. It’s a lot of work, but the rewards are definitely worth it.

Over time their sandbox databases, development pools and other workspaces become crucial components of the aCRM set of data sources, suddenly requiring enterprise support, maintenance and reliability. However, due to the historically grown anarchical structure only very adventurous IT departments would be willing to conclude an SLA for such systems, which are hard to disentangle and maintain.

At first sight there seems to be no escape from such a road block. Depending on the balance of power, either BI will insist on a solid evolution of its systems and even enforce an internal monopoly, which imposes their rhythm on all internal customers. Or the power users extend the corporate BI systems with the many little things they need and value while progressing steadily towards an unmanageable situation. Compliance becomes a sheer nightmare. Collaboration between marketing and IT often unpleasant.

Either way, be it the freedom for power users approach or the IT rules approach, it is not sustainable. Flexible processes need to enable the business to obtain results at a speed acceptable for them and handle the transition between prototyping systems towards solid ones whenever appropriate. Throwaway extensions which have been used once or twice and left alone afterwards, should be flagged as such, archived and phased out of the workspace in order to avoid that such dead bodies prevent you from taking care of the extensions still being used.

But from time to time one or more of these disposable graphs turn out to be nuggets — valuable discoveries or insight that deserve being shared with a wider audience. So the graph — say a histogram comparing numbers of contacts for a few segments — is already there. Capturing this by-product should not slow us down on our way towards the model we are heading for. Unfortunately the histogram’s layout, axes and captions are not in compliance with the visualisation standards and rules we have been preaching and teaching before^[2] . The histogram with tick marks for 0.8  1.8 and so on until 19.8 for these difficult to be more integer data is simply not presentation ready. Too bad.

Now starting up another tool reproducing the selection, the little transformation and the histo itself will likely take a few minutes and distract us from our core task, the model to be built.

So after all, we might be inclined to limit the celebration of the nugget to showing it to their colleague who happens to take a coffee 2 meters away. But anything more would be too much hassle and not as target oriented as we ought to be. At least as long as data mining suites and presentable graphics remain two different stories.

PS: This hold actually more for Clementine and KNIME which have the aspiration of being exhaustive and fully integrated. SAS will often require the use of two tools (Guide and Miner) anyway, depending on where the data come from.

[2] See for example http://www.perceptualedge.com/blog/ or various publications on data visualisation for guidelines how presentable graphs should look like.