Slowly, more and more analyst firms and vendors draw a more differentiated picture and accept that there is a large range of user situations and needs which ask for a range of appropriate technical answers.

One of the most important goals of the data warehouse is to provide a single version of the information. In order to avoid potentially different data selections or transformations it would only be sensible to also have just one tool to access the single version of the information, eradicating the problem of different numbers for the same question once and for all. This might be the strongest motivation for BI professionals and business users who seek a universal BI end user tool that answers all business needs and widens the access bottle neck. The growing market opportunity for BI tools motivates the software vendors to promise technological breakthroughs and even miracles where required.

The challenge becomes apparent when taking a closer look at what different users need. At least three use cases need to be distinguished and give rise to different and conflicting software requirements:

• „Most data warehouses will probably form the basis for a stream of ready-made reports being send out to many users across the organisation. Usually no interactivity is provided with the reports and most users would more likely become confused by features allowing for customisation. What is needed are capabilities to automatically and repeatedly generate reports combining several tabular and graphical elements.
• „Senior management wants the convenience to oversee the entire business and drill down into subject areas when indicated. This need can be answered by information dashboards providing basic flexibility via an intuitive GUI. The bigger challenge here is to select and define the pieces of information to be displayed in line with the rest of the organisation.
• „Follow-up questions triggered by the two first user groups and all other analysis give rise to non-standard or ad hoc queries. They involve not only tools with full query flexibility but most importantly data analysis skills and knowledge of the data sources restricting these queries to a small minority of the users for most organisations.  Data warehouse thought leader Ralph Kimball wrote in 1998: “Ad hoc query tools, as powerful as they are, can only be effectively used and understood by about 10% of all the potential end users of a data warehouse. … The very best ROLAP-oriented ad hoc query tools improve the 10% number to perhaps 20%.” ^[1]
  Also data discovery tools, data mining workbenches and even plain SQL tools fall in this broad category.
  See also Stephen Few’s article Big BI is Stuck: Illustrated by SAP BusinessObjects Explorer (his Blog is highly recommended in general).

These three use cases could be further broken down, for example, by distinguishing between being OLAP focused or not, and vendors tend to showcase certain capabilities (such as data search) as constituting a whole new category.

It shows that a unified tool for all users and situations could probably only be developed at the expense of too many compromises and would probably satisfy no one in the end.

The even more utopian plan of BI delivery by pure self-service could only work with a complete data warehouse presented by a clearly documented user interface layer and in situations where queries are limited to simple look ups of stored facts. I’m not sure that the three conditions are fulfilled anywhere.

He also mentions, as I have in a previous article here, that master data management (MDM) systems can be used to remove from the traditional data warehouse the complex work of data integration. I see this happening more and more for the reasons I expressed earlier, mainly a reduced total cost of ownership and an increased usability of the information. MDM systems are perfectly suited to the “data integration” part of the overall business intelligence world, and as such, helps to allow the traditional data warehouse evolve to a smaller, quicker, and less expensive machine to operate. Also, sourcing a data warehouse from an MDM system means that analytical data can be provided far easier and quicker than before, and this supports Neil Radan’s BI 2.0.

I have worked with organizations that have come to the conclusion that the types, amounts and sources of information needed to run the business properly are constantly changing as the organization tries to keep up with or keep ahead of the competition in a more and more volatile business environment. What resulted was the organization stepping backward in the maturity of its data warehouse (according to the ” Data Warehouse Maturity Model” I mentioned in a previous article). One organization in particular, to get the data the business needs in front of them faster, did an end-run around the data warehouse and started pulling data from whatever source system they thought had what they needed. I have to admit that the web-based interface they built was very impressive, and they could indeed deliver information quickly.

The problem I had with it was simply that they had ignored all the “best practices” about doing good data integration, good data governance and good data management. Because they built their solution outside the normal project path of the organization, in order to avoid the lengthly project schedules and extra costs, they side-stepped everything the organization had put in place to mitigate just the kinds of problems that this new development brought up again. And the organization had suffered the same problems in the past! I was amazed that the business unit in question was allowed to do this, but in the end, you had to sympathize with them. All they wanted was the data in a form they could use, and quickly. The solution should have been to change the processes to allow for these kinds of tools, but still keeping the principles of proper data management in place.

So, they were trying to do Business Intelligence 2.0 by using Web 2.0 type features on top of an immature approach to enterprise data management. In the end, I think it will hurt them more than help them, but who knows. These days things change so fast inside and outside business’s that even problems like this can disappear amongst everything else that goes on.

I would like to know if anyone is or has worked somewhere where a proper implementation of “Business Intelligence 2.0″ is or  has happened. Please share your experiences in the comments section.

All of that reminded me of the Data Warehouse Maturity Model I have seen time and again at conferences I have attended and presented at. It shows that a data warehouse has reached its ultimate maturity when it achieves its potential for the return on the investment of building a data warehouse. That is, the business uses the data warehouse and the information within it to drive the business. I started to think about this while I was putting the business case together for this project. Too often, I have found that the decision makers in an organization are not always aware of the value of the data warehouse, and see it as a big, expensive, slow moving giant that they have to live with rather than really benefit from. It is beyond the scope of this article to discuss all the reasons why this might become the perception of the data warehouse, but it was sure the case here, which is why I needed to include in my business case the feedback from all the users of the data warehouse.

The organization in question was looking for ways to reduce the costs of the data warehouse and it occurred to me that one way to reduce the cost of it was to remove it all together from the business. I don’t mean throw it away (that would be insane), but why not get a little creative? Take all the hardware, the software, and the people that build, operate and maintain it all, and use it to spin off a separate business? This business would be initially funded by the parent organization, contracted by the parent to provide all the data warehouse services it did before but at a more favourable cost. It would operate as an independent business with the sole core competency of doing data warehousing, and be allowed to also sell its services to other organizations as well. By being wholly owned by the parent company and being able to expand its customer base beyond the parent, allows it to become a revenue centre for the parent. Eventually the separation would probably need to be more complete, I suppose, but still, it goes from being a cost centre to a revenue centre. Sweet!

Afterwards, I thought, hey, I’ve been doing software design and development for over 20 years, data warehousing for almost 10, and ran my own small consulting firm. Why don’t I just start up a business that provides a data warehouse as a service? Why, I thought to myself, that’s a great idea and would be a lot of fun! I know a couple of organizations that are moving in the direction of having other “systems as a service” anyway (at least outsourcing other major business support systems), so I could probable lock up a customer or two within a couple of years, and who knows? I started to seriously investigate the possibility. I would need to build/buy/rent a data center and build/rent/lease a bunch of servers and disk, not really a problem, but I would (ideally) need to come up with a data architecture that could support multiple clients, possibly across different industries. That would mean, design it, build it, test it, implement it, and then go sell it. Now that’s a lot of work. And I know that you can buy such industry specific data models. Teradata did, or maybe still does, have one, Oracle has one to, and I am sure there are others. Possible still, but a lot of work, and it would mean I might miss the opportunities I had because of the timing.

In continuing my investigation into setting all this up, I came across a company that already provides Data Warehousing as a Service (DaaS), called Kognitio. They have their own data centres, they have their own data models, and although they are the only ones I could find, but there might be others by now. So, someone had my genius idea before me (no surprise really). I still think that for an organization that is trying become more mature in their data warehousing, this is a good solution to get off the ground fast and jump the “gulf between ‘Inform Exectutives’ and ‘Empower Knowledge Workers’” in the afore-mentioned data warehouse maturity model. If you are looking to implement a new data warehouse, or re-architect one, or try to out-source you existing one, I suggest you give them a call. I think I might, just to see if they need an affiliate in Switzerland, where I live.

What do you thing of Data Warehouse as a Service? Good idea or bad? Easy or difficult to implement? Too big a change to your organization or not? Let us know your thoughts in a comment!

The Solutions

To address the challenge of maintaining up-to-date, complete and accurate technical documentation of the data warehouse platform, CAST was used to analyze the production platform after every deliverable. The Enlighten tool was made available to all the internal and external staff working on data warehouse projects, and the web-site created by CAST was made available to the entire organization.

And because we also had several on-going projects  (as everyone does), we would also analyze the UAT environments from time to time as well. This allowed the impact analysis to capture changes to a future version of the data warehouse platform as well as the current one.

The code quality analysis was reviewed on a periodic basis to keep track of the “hotspots”, as these problem areas would be included in projects that touched on those areas, so that the number of potential problem areas could be reduced over time.

The Benefits

By using CAST, we were able to reduce the costs of project affecting the data warehouse while increasing the quality of the deliverables. These cost saving were realized by having all the technical information of the platform in one place, always up-to-date, and easily available and navigatable. As well, by automating many of the otherwise manual tasks that never seem to get done, the cost of testing and doing impact analysis was signicantly reduced. Also, the abilty to set the expectations of quality and being able to measure those expectations objectively, let to better, more bug-free code being delivered on-time and under budget.

Recommendation

I highly recommend this product to anyone who has a large system or systems that they need to get under control or not lose control of. CAST helps to implement a layer of automated governance that can be otherwise far more expensive to implement, and, as mentioned above, can certainly reduce project costs through increased productivity and increased quality of deliverables. I haven’t worked on a project since where I didn’t wish CAST had been implemented there as well.

In the next post, I will look at some things that we could have done with CAST to further improve our business processes, and a couple of things to take into consideration when using CAST. If you have any questions, I urge you to contact them, and you can also leave questions in the comments below and I will answer the to the best of my ability.

I recently worked with an organization where we had looked at a tool that was able to analyze all the source code for the ETL jobs of their data warehouse and provide several benefits to the organization as a result. The tool is called CAST, and comes from a company called Cast Software.  I was hesitant to implement such a tool there because I felt that the data warehouse group didn’t operate at a sufficient level of maturity to be able to fully exploit the features of this software, and as a result, would be implementing a tool that would never get used. However, during the presentation of the tool and its features by the CAST team, I, for one, was so impressed by what the tool could do in terms of analyzing source code, and even more, what it does with the information gathered from that analysis, that we decided to implement the tool. In the end, the data warehouse group was able to take advantage of the tool, although not as completely as I would have liked. In the next few articles, then, I will examine in more detail, why we decided to implement the tool, what it could do out of the box, what it couldn’t do out of the box (but could do later, with a little customization), what we did with it, and what we didn’t do with it but probably should have.

The Challenges

Because our data warehouse had been built in bits and pieces over a number of years, there was a lack of cohesive technical documentation. It existed, of course, but in documents spread over hundreds of project directories, and in the heads of the internal staff and in the heads of the external staff, if they were still there. This was always a problem in that, a) they were not sure what they had, or how it was built, b) doing an impact analysis across all the ETL tools involved (unix shell scripts, DataStage, custom PL/SQL, and some java) was very labour intensive, and c) with all the ongoing projects for the data warehouse, it was impossible to properly evaluate the quality of the deliverables from the vendors (most importantly, the source code).

Getting More Done Better With Less

We originally looked at CAST to solve the above problems. The tool could analyse all the source code across all the ETL tools we had, and in doing so, could provide a) complete technical documentation of the production data warehouse platform, b) end-to-end impact analysis, and c) provide a quantitative analysis of the source code delivered by the vendors and provide industry-standard measures of the quality of the code. Having all this would, in turn, allow us to a) significantly reduce the learning curve for new internal staff and external vendors, b) significantly reduce the time and increase the quality of the impact analysis studies they carried out, and c) enable automated enforcement of the development standards and guidelines, reducing the number of errors in the deliverables whilst improving the overall code quality.

Code Analysis

CAST can parse just about all the source code you can throw at it, regardless of the language or tool it was written in. If they don’t already have a parser written for a specific tool, their universal parser can be tweaked very quickly. With our organization, we had some source code in Unix shell scripts, some ETL written using DataStage, and some just in PL/SQL. The reporting tool used was Business Objects. CAST had no trouble with anything we gave it. The one issue we did have was that the PL/SQL parser was not equipped to deal with dynamic SQL and PL/SQL that was data driven. Much to my surprise, though, it took the CAST Switzerland team only a handful of days to update their PL/SQL parser to handle this problem! When they were done, we had a tool that could analyze all the source code in our platform, along with the databases (mainly Oracle) so that we ended up with complete technical documentation (albeit using the included Enlighten tool, with a bit less on the generated web pages). To put it succinctly, they managed to do in 12 days, what we had not been able to do in almost 10 years!

Impact Analysis

With all the source code analyzed, and all the database objects analyzed, the CAST Enlighten tool could generate an automated impact analysis in the time it would take to get a coffee before starting to do the analysis manually. In the past, doing a thorough impact analysis required extensive reviews of document when you could find them, reading all the source-to-target documents if they were up-to-date, talking to the developer if they were still on-site, and in the end, reading all the source code line by line if you could get access to it. As you can imagine, this was not only time-consuming, but frought with errors and omissions. It could take weeks on a larger change, but with CAST, it could be done in an afternoon, with just a little digging inside the tool itself.

Code Quality

Another big issue we suffered with was that there were never the resources within the team to do a proper inspection and review of the source code deliverables from the vendors. In some cases, the projects were just too big to manage with our small internal staff size, in other cases just too small to worry about. This led to visicous, lengthly cycles of testing and re-developing, as new bugs were not found until user acceptance testing, or, even worse during the pre-production system testing, and in a few cases during use after production rollout. Not to mention the inevitable commercial disagreements between the vendors and us where one would blame the other for the delays and the errors. Since we didn’t have the internal resources to do this due-diligence of the vendor deliverables, a lot of problems like the ones mentioned above crept into every project, pushing back delivery schedules and blowing the budgets of the projects.

The CAST tool, because it comes with a huge number (>600 when we used it) of quality measures for source code, and because you can modify them to your taste or add new ones, all the delivered source code could be checked against these quality measures, even before they were manually tested. I didn’t pretend to understand all the types of measures they would calculate against the source code (things like Cyclomatic Complexity are the things of rocket scientists, I’m sure), but what I did understand was that they could tell you in an instant how well written the code was, how well it complied with your internal standards as well as industry standards, and where in the code you might have a trouble spot later on (if the complexity measure was too high). All these things could be used to do an automated reinforcement of the development standards and guidelines of the organization, and acted as an independant third-party to help in the decision to accept or not the source code being delivered by a vendor.

continued in the Part 2…

Isn’t it time to restrict the obedience to the business side deciding about what is being developed? Shouldn’t IT considerations at least be as seriously considered when it comes to how to deliver the right systems and quickly? Agile methods are particularly appropriate for BI systems which are close to the end users. That users often cannot precisely formulate their needs initially, is one of the strongest arguments for an incremental and iterative process. But there are aspects specific to BI favouring prototyping. Encouraging active user involvement throughout the development is certainly of high value for both sides. Delivering the parts of highest priority first and using that output can give very useful input for subsequent phases including the option of stopping the project, if first results made the project obsolete or changed the requirements drastically. Among the strongest critisms BI teams face are being slow and not delivering very useful results. Both issues have been strong motivations for agile approaches since the beginning.

As Wikipedia puts it, agile development is “based on iterative development, where requirements and solutions evolve through collaboration between self-organizing cross-functional teams”. It is in particular “a business approach that aligns development with customer needs and company goals.”

We might need to reinterpret some of the principles of the agile manifesto. The principle working software is delivered frequently might sometimes be read results are delivered frequently. It is the close collaboration between the BI team and the users supported by frequent output to the users keeping them up to date that matters here. A principle that seems very hard to swallow and contrary to most IT development culture says that even late changes in requirements are welcomed. Ok, maybe not all changes are appropriate and need to be expressed so late. But this openess and flexibility is key to agile development. It means accepting that it is hard to impossible for users to sign and close BI requirements before having seen the data.

The fact that relatively few organisations apply agile processes probably does not mean that people are all happy with the waterfall process. Many large organizations have difficulty bridging the gap between the traditional waterfall method and an agile one. It would also require getting around reasons predominantely external to BI such as overall budgeting, planning and roadmap management and prioritisation.

Wikipedia describes the agile home ground as projects of low criticality with a small team of senior developers and requirements that change very often. A consultative management culture would help.

Don’t get me wrong. Major infrastructure projects, be it enterprise data warehouse or something operational, are not good initial candidates for process innovation. Where the big picture is important to be kept in mind, where the long term maintenance and flexibility has to be assured and users interact with the deliverable only indirectly, a classical planned approach still has its virtue. But when it comes to small analytical marts and BI interfaces we should take a good dose of self-confidence and go agile even it takes a few serious discussions.

What I mean is that in the big picture of enterprise data management, there are data applications that produce data, and data applications that consume data. In a lot of cases, a data application can (and must) play both roles. I make the distinction here to show one of the great promises of master data management: the reduction in the amount of duplication of work being done by software processes that must transform data between the applications that produce that data, and the applications that consume it. In the following diagram, we can see what can happen if several data consumer applications (right side) must connect directly to other data producing applications (left side) to get (and transform) the data they need.

Not a Pretty Picture.

As you can see, it can get pretty messy, pretty quickly. There are things that exist in the IT world to try and reduce this complexity, SOA, EAI, etc. and some do a reasonable job, but in the end, there is more needed to really remove the duplication of effort. This is master data management (MDM). If we have an MDM hub in the middle of the above diagram, to act as a level of abstraction between data producers and data consumers, we can significantly simplify the picture and remove all redundancy.

Much Better.

The MDM hub works the most effectively, where there is an actual data store implemented within it. The efficiency arises from being able to separate source-system specific data transformations from source-system independent data transformations. It enables this by maintaining, at its core, a enterprise-level business entity view of the data held in the department level (purpose-specific) data applications. This business view of the data, is populated from the source system’s data using source-system specific transformations, and thereby transforming the data into a data model that represents how the business sees the data entities, rather than how the purpose-specific applications see the data. When the data is transformed in this way, into an enterprise level business view of the data, the enterprise as a whole can understand the data and use the data by doing further transformations of that data into purpose-specific data (i.e. data to improve marketing, customer care, logistics efficiency, etc.).  The beauty of the whole system is that the end-users of the data do not need to worry about directly transforming the source system data into their own purpose-specific data, especially when they might need the same type data from multiple source systems (each with their own way of storing that information). They only need to go to one place, where the structure of the data is oriented to how they see the data, and where the data means the same thing thing to everyone (enterprise level). They are then isolated from the changes underneath, that is if a source system is replaced, decommissioned, or radically changed, the impact is kept to a minimum. The only changes required are the source-system specific transformations, and nothing else.

To get back to my original argument, I believe that an MDM system is mandatory for efficient operation of any medium to large-sized organization, and gets more important as the enterprise gets bigger. And I like to keep the main roles of the data managers separate from one another (at least on a personnel level, it not at an organizational structure level). That is, I prefer to have those individuals who use the data to run the business (data consumer enterprise roles) focused on using the data, and those individuals who are responsible for producing the data, focused on the applications that produce the data, regardless of where they fit into the organization’s operational structure. Therefore, I maintain that those individual who are responsible for the MDM system focus on doing that to the best of their abilities, and those that use the MDM system, such as those who would use the MDM system as the main source for the data warehouse, focused on building and maintaining the data warehouse.

So, at the end of the day, the bulk (possibly) of your data integration would happen in the MDM system, using source-system specific transformations, which would then allow the data warehouse to focus on generating analytical data for downstream use , without having the added burden of being responsible for the data integration as well, having only to worry about using source-system independent transformations, like many other downstream applications.

The ultimate result of the data warehouse team having to do this reconciliation on an on-going basis, is the promotion of the idea of data governance at the enterprise level. Pushing the ownership of the meanings (thus the definitions of) the data within the data warehouse, back to the organization seems like the right thing to do, and of course, it is. It is, if it’s done properly, because what we really want to do by implementing some data governance in the organization, is to implement a master data management system, which data governance is an important part of. We are saying, “We know that the data must be managed at the enterprise level, not at the department level, and we must engage the entire enterprise in the management of the data.”.

The management of “data” at an enterprise level means removing the shackles of departmental ownership of the data while still maintaining the departmental systems that manage that data. This is, by far, the trickiest part, as it deals with people, rather than just the technology and we will examine this in a later post. For now, though, we need to understand the implications of what it means to “manage data at an enterprise level”. These implications include

• restructuring the organization a little to implement enterprise data governance,
• looking at, planning for, and possibly phasing in a master data management system,
• removing some or all of the responsibility for data integration from the data warehouse (controversial, I know),

to mention but a few. We will go into more details about these and other issues in future posts as well.

Becoming more mature as an enterprise in how we deal with our data requires a shift in the way we look at, work with, and manage the data that we need to run our enterprise. It’s a lot of work, but the rewards are definitely worth it.