9 rock-solid arguments for a data warehouse

What is the use of a data warehouse?

More and more organizations wonder aloud what the use of a data warehouse is and whether it is worth the investment. They also want to know which alternatives exist. A growing number of IT vendors and some so-called experts claim that the end of the data warehouse is in sight. These are vendors of data warehouse appliances, data virtualization tools and data discovery tools.

We are convinced otherwise. The data warehouse is still the beating heart of every intelligent, data-driven organization. It fulfils nine vital functions.

A data warehouse is a separate environment in which data from different source systems is integrated, kept with its history and structured for fast reporting and analysis.

1. Improving integration

An organization records data in different systems that support the different business processes. To create an integrated picture of operations, customers and suppliers, you have to bring the data together in one place and make it compatible. That also creates one version of the truth. Both external data (from the environment) and internal data (from ERP, CRM and financial systems) flow together in the data warehouse and are neatly grouped, so that end users can effortlessly combine and analyze data from different sources.

2. Speeding up response times

Source systems are fully optimized to process many small transactions in a short time: adding orders, applications or reports. For management information about the performance of the organization you send only a few large “transactions” to your data sources, such as the revenue figures for the past twelve months, which involve millions of records. You want to collect, aggregate and display a great deal of data at the push of a button. The structure of a data warehouse is specially designed to retrieve and analyze such large quantities of data quickly.

3. Faster and more flexible reporting and analysis

The structure of a data warehouse enables end users to report quickly and flexibly. With the right tools they can also carry out interactive analyses from different predefined angles, called dimensions. Users can jump from year to quarter to month with a single click and switch quickly between the customer and product dimension, while keeping the same indicator. In this way they can really juggle with the data, quickly gain valuable knowledge about operations and customers and monitor the scores on performance indicators.

4. Recording changes and building up history

Many source systems do not keep the history of crucial records properly. When a customer moves, or when a product manager transfers a product to another product group, the system probably just overwrites the old values, and then you have lost them. The “memory” of the source system disappears.

To generate reliable information you need those old values, so that users can look at the performance of the organization from a historical perspective: in accordance with the organizational structure, customer locations and product classifications of that time. If they can only view the data from the current context, they miss crucial information. A data warehouse automatically retains the changes in your data sources, which makes historical analyses very simple.

5. Raising the quality of data

Organizations often overestimate the quality of the data in their source systems, until they start browsing through the data and see with their own eyes that much of it is incorrect or incomplete. With a data warehouse they can improve data quality step by step and structurally. Errors are eliminated, either by correcting the data automatically during loading, or by smart detection followed by tackling the quality problem at the source.

6. Relieving the operational systems

By transferring data to a separate environment, organizations relieve the operational systems, which could go down under extensive or complex analyses. Thanks to a data warehouse the source system can keep fulfilling its primary role at all times. Operations continue undisturbed while users make the most complex analyses, and the historical, integrated analysis environment can be fully optimized for its task.

7. Unburdening the IT department

A data warehouse and user-friendly Business Intelligence tools enable employees to make reports and carry out analyses independently. An organization does first have to invest in the required infrastructure. The better the architecture is set up and developed, the easier it is for users to produce complex reports and analyses independently of IT.

End users of course first need sufficient training and support. In practice the BI department often still makes many of the more complex reports itself, because users lack time or knowledge of the process, because the structure of the data sources is too complex or because they find the tool too difficult. Another reason may be that too little effort has gone into the right architecture. That is often the bottleneck: setting up and maintaining the architecture takes time, expensive data engineers and ongoing management. Fortunately a large part of that repetitive engineering work can now be automated, for example with Diamond DWH.

8. Increasing recognizability

Indicators are “pre-baked” in the data warehouse as much as possible, including the more complex, genuine KPIs. This enables users to produce more complex reports in a few simple steps, for example a report with the return at customer level broken down by month, customer group and country. In the source system you can only make this information by carrying out a large number of manual actions and calculations and by adding data from other sources.

A data warehouse thus increases the recognizability of the information decision makers need to steer the organization. This only holds if the data warehouse is set up with the business as the starting point and forms a mirror of the business processes. Ideally you therefore do not start with the source systems but with the decisions you want to improve: a decision-first approach.

9. Increasing findability

Setting up a data warehouse also gives users quick access to the meaning and origin of data (metadata). In many source systems, certainly in the cloud or from specific vendors, these meanings are not always easy to find, and sometimes very poorly accessible. With a data warehouse users find the right data sooner and work it up into usable information, insights and knowledge.

Reporting on source systems versus a data warehouse

The nine reasons summarized in five differences.

Conclusion

The nine reasons make clear why a data warehouse is still the beating heart of a data-driven organization. Alternatives such as appliances or data virtualization can be a partial solution, but do not match the total solution a data warehouse offers. And the choice is not all or nothing: fast where possible, thorough where necessary.

Further reading

• Data warehouse
• Data quality
• Blog: Power BI and the data warehouse
• Blog: 6 reasons not to build a data lake
• Blog: The pros and cons of a data vault

About the author

Dick Pouw obtained his MBA at Business School Nederland. He has more than 20 years of experience as a change manager in customer care, IT and business intelligence, including as an interim line and transition manager in the semi-public, retail and media sectors.

About Diamond Decisions

Diamond Decisions helps organizations become better by enabling better decisions. We don’t start with data, dashboards or AI, but with the decision that needs to improve, and work back from there to KPIs, data and technology. We do that in four steps: inspiration and alignment, business modeling, analytics and data platform and data-driven decision makers. That way BI and AI don’t remain an IT project, but deliver value every day.