1. The ability to create ERD diagrams or at least a facsimile thereof using UML class diagrams. Extra points awarded if it does both or if the UML class diagrams can be customized for ERD type diagrams
2. The ability to generate web-sites from the final diagrams and/or any other supplementary information that can be added to the diagram objects
3. Speaking of which, extra points awarded for being able to attach significant amounts of meta-data and/or reference external files
4. Naturally, ease-of-use will be a priority
5. As will a short learning curve
6. Ease of implementation is a bonus as will be the ability to install the tool on multiple platforms (in my case, I will test some version of windows, Snow Leopard (Macs) and Linux (likely Ubuntu).

I will also include whatever pains I went through during the installation as well, if there was any. Since I run mainly Snow Leopard and will need to virtualize environments for Windows (probably running Wine under OS X) and Linux, I am guessing there might be some. I will keep the pain of installing in the virtual environments separate from the pain of installing the tool itself (which there really shouldn’t be any).

If you have any particular data modelling tool that you would like me to test, please leave a comment below. Also, soon, I will present a compiled list of contenders, so speak up quickly .

He also mentions, as I have in a previous article here, that master data management (MDM) systems can be used to remove from the traditional data warehouse the complex work of data integration. I see this happening more and more for the reasons I expressed earlier, mainly a reduced total cost of ownership and an increased usability of the information. MDM systems are perfectly suited to the “data integration” part of the overall business intelligence world, and as such, helps to allow the traditional data warehouse evolve to a smaller, quicker, and less expensive machine to operate. Also, sourcing a data warehouse from an MDM system means that analytical data can be provided far easier and quicker than before, and this supports Neil Radan’s BI 2.0.

I have worked with organizations that have come to the conclusion that the types, amounts and sources of information needed to run the business properly are constantly changing as the organization tries to keep up with or keep ahead of the competition in a more and more volatile business environment. What resulted was the organization stepping backward in the maturity of its data warehouse (according to the ” Data Warehouse Maturity Model” I mentioned in a previous article). One organization in particular, to get the data the business needs in front of them faster, did an end-run around the data warehouse and started pulling data from whatever source system they thought had what they needed. I have to admit that the web-based interface they built was very impressive, and they could indeed deliver information quickly.

The problem I had with it was simply that they had ignored all the “best practices” about doing good data integration, good data governance and good data management. Because they built their solution outside the normal project path of the organization, in order to avoid the lengthly project schedules and extra costs, they side-stepped everything the organization had put in place to mitigate just the kinds of problems that this new development brought up again. And the organization had suffered the same problems in the past! I was amazed that the business unit in question was allowed to do this, but in the end, you had to sympathize with them. All they wanted was the data in a form they could use, and quickly. The solution should have been to change the processes to allow for these kinds of tools, but still keeping the principles of proper data management in place.

So, they were trying to do Business Intelligence 2.0 by using Web 2.0 type features on top of an immature approach to enterprise data management. In the end, I think it will hurt them more than help them, but who knows. These days things change so fast inside and outside business’s that even problems like this can disappear amongst everything else that goes on.

I would like to know if anyone is or has worked somewhere where a proper implementation of “Business Intelligence 2.0″ is or  has happened. Please share your experiences in the comments section.

All of that reminded me of the Data Warehouse Maturity Model I have seen time and again at conferences I have attended and presented at. It shows that a data warehouse has reached its ultimate maturity when it achieves its potential for the return on the investment of building a data warehouse. That is, the business uses the data warehouse and the information within it to drive the business. I started to think about this while I was putting the business case together for this project. Too often, I have found that the decision makers in an organization are not always aware of the value of the data warehouse, and see it as a big, expensive, slow moving giant that they have to live with rather than really benefit from. It is beyond the scope of this article to discuss all the reasons why this might become the perception of the data warehouse, but it was sure the case here, which is why I needed to include in my business case the feedback from all the users of the data warehouse.

The organization in question was looking for ways to reduce the costs of the data warehouse and it occurred to me that one way to reduce the cost of it was to remove it all together from the business. I don’t mean throw it away (that would be insane), but why not get a little creative? Take all the hardware, the software, and the people that build, operate and maintain it all, and use it to spin off a separate business? This business would be initially funded by the parent organization, contracted by the parent to provide all the data warehouse services it did before but at a more favourable cost. It would operate as an independent business with the sole core competency of doing data warehousing, and be allowed to also sell its services to other organizations as well. By being wholly owned by the parent company and being able to expand its customer base beyond the parent, allows it to become a revenue centre for the parent. Eventually the separation would probably need to be more complete, I suppose, but still, it goes from being a cost centre to a revenue centre. Sweet!

Afterwards, I thought, hey, I’ve been doing software design and development for over 20 years, data warehousing for almost 10, and ran my own small consulting firm. Why don’t I just start up a business that provides a data warehouse as a service? Why, I thought to myself, that’s a great idea and would be a lot of fun! I know a couple of organizations that are moving in the direction of having other “systems as a service” anyway (at least outsourcing other major business support systems), so I could probable lock up a customer or two within a couple of years, and who knows? I started to seriously investigate the possibility. I would need to build/buy/rent a data center and build/rent/lease a bunch of servers and disk, not really a problem, but I would (ideally) need to come up with a data architecture that could support multiple clients, possibly across different industries. That would mean, design it, build it, test it, implement it, and then go sell it. Now that’s a lot of work. And I know that you can buy such industry specific data models. Teradata did, or maybe still does, have one, Oracle has one to, and I am sure there are others. Possible still, but a lot of work, and it would mean I might miss the opportunities I had because of the timing.

In continuing my investigation into setting all this up, I came across a company that already provides Data Warehousing as a Service (DaaS), called Kognitio. They have their own data centres, they have their own data models, and although they are the only ones I could find, but there might be others by now. So, someone had my genius idea before me (no surprise really). I still think that for an organization that is trying become more mature in their data warehousing, this is a good solution to get off the ground fast and jump the “gulf between ‘Inform Exectutives’ and ‘Empower Knowledge Workers’” in the afore-mentioned data warehouse maturity model. If you are looking to implement a new data warehouse, or re-architect one, or try to out-source you existing one, I suggest you give them a call. I think I might, just to see if they need an affiliate in Switzerland, where I live.

What do you thing of Data Warehouse as a Service? Good idea or bad? Easy or difficult to implement? Too big a change to your organization or not? Let us know your thoughts in a comment!

What I mean is that in the big picture of enterprise data management, there are data applications that produce data, and data applications that consume data. In a lot of cases, a data application can (and must) play both roles. I make the distinction here to show one of the great promises of master data management: the reduction in the amount of duplication of work being done by software processes that must transform data between the applications that produce that data, and the applications that consume it. In the following diagram, we can see what can happen if several data consumer applications (right side) must connect directly to other data producing applications (left side) to get (and transform) the data they need.

Not a Pretty Picture.

As you can see, it can get pretty messy, pretty quickly. There are things that exist in the IT world to try and reduce this complexity, SOA, EAI, etc. and some do a reasonable job, but in the end, there is more needed to really remove the duplication of effort. This is master data management (MDM). If we have an MDM hub in the middle of the above diagram, to act as a level of abstraction between data producers and data consumers, we can significantly simplify the picture and remove all redundancy.

Much Better.

The MDM hub works the most effectively, where there is an actual data store implemented within it. The efficiency arises from being able to separate source-system specific data transformations from source-system independent data transformations. It enables this by maintaining, at its core, a enterprise-level business entity view of the data held in the department level (purpose-specific) data applications. This business view of the data, is populated from the source system’s data using source-system specific transformations, and thereby transforming the data into a data model that represents how the business sees the data entities, rather than how the purpose-specific applications see the data. When the data is transformed in this way, into an enterprise level business view of the data, the enterprise as a whole can understand the data and use the data by doing further transformations of that data into purpose-specific data (i.e. data to improve marketing, customer care, logistics efficiency, etc.).  The beauty of the whole system is that the end-users of the data do not need to worry about directly transforming the source system data into their own purpose-specific data, especially when they might need the same type data from multiple source systems (each with their own way of storing that information). They only need to go to one place, where the structure of the data is oriented to how they see the data, and where the data means the same thing thing to everyone (enterprise level). They are then isolated from the changes underneath, that is if a source system is replaced, decommissioned, or radically changed, the impact is kept to a minimum. The only changes required are the source-system specific transformations, and nothing else.

To get back to my original argument, I believe that an MDM system is mandatory for efficient operation of any medium to large-sized organization, and gets more important as the enterprise gets bigger. And I like to keep the main roles of the data managers separate from one another (at least on a personnel level, it not at an organizational structure level). That is, I prefer to have those individuals who use the data to run the business (data consumer enterprise roles) focused on using the data, and those individuals who are responsible for producing the data, focused on the applications that produce the data, regardless of where they fit into the organization’s operational structure. Therefore, I maintain that those individual who are responsible for the MDM system focus on doing that to the best of their abilities, and those that use the MDM system, such as those who would use the MDM system as the main source for the data warehouse, focused on building and maintaining the data warehouse.

So, at the end of the day, the bulk (possibly) of your data integration would happen in the MDM system, using source-system specific transformations, which would then allow the data warehouse to focus on generating analytical data for downstream use , without having the added burden of being responsible for the data integration as well, having only to worry about using source-system independent transformations, like many other downstream applications.