Data mesh cuts into the central BI team and the data warehouse

What is a data mesh?

A data mesh is an architecture in which you largely build your business intelligence locally, decentrally at domain level, package the results as a fully fledged quality product and deliver that product to users in other business domains, for both transactional and analytical purposes.

To clear up a misunderstanding at once: according to Rick van der Linden a data mesh is not a concrete product. No special software tool belongs to it that you can order from a supplier, although some suppliers would have you believe otherwise.

Where does the data mesh come from?

According to the inventor of the concept, Zhamak Dehghani, formerly of technology company Thoughtworks, a data mesh is a decentralized data architecture in which the business domains own their data. They themselves are responsible for access, storage, performance, quality and delivery of data products. In a federated model they autonomously guard data quality and security and take care of governance. On the basis of a self-service model, domains request the data they need from the other domains, no longer from a central business intelligence department. Dehghani developed it as a third-generation data platform, an alternative for organizations that get stuck with a central data warehouse or a data lake.

That does not happen by itself, says Van der Linden. “As in the real world, a federation involves a lot of coordination, communication and politics. Think of the United States, the Swiss cantons or the Federal Republic of Germany. In a data mesh too you have to get everyone on board.”

Data mesh tackles poor data quality at the source

Van der Linden: “In my view a data mesh finally offers a solution for the poor data quality organizations have been struggling with for decades. I do not believe in a central department that guards data quality. It leads too often to discussions: which data source is leading and why are there always two versions of the truth?” If you are a data manager in the HR domain, you are responsible for the quality of all personnel data, and you must be able to offer that data as a decent product to everyone who needs it. If everyone keeps to the agreements on metadata, definitions, delivery specifications and communication protocols, it can no longer happen that, through all kinds of copying and timing differences, the number of access passes issued does not match the number of employees in service.

Bridging the gap between operational and analytical data

In her groundbreaking articles Dehghani identifies a gap that has arisen between operational (transactional) data and analytical data. The analytical discipline makes different demands: data is retrieved on characteristics rather than keys, and always in large quantities at once. Because the data mesh prescribes that both transactional and analytical products are produced locally, in the same place, that gap disappears.

Data mesh simplifies the plumbing

The gap has also led to a fragile architecture. ETL tasks stagnate because lead times keep growing as a result of large data volumes, a huge number of sources and overloaded hardware. A labyrinth of data pipelines has arisen. The data mesh reduces the complexity by dividing the labyrinth into planes, in which each domain takes up its own tasks. Van der Linden: “Architecture is, put bluntly, plumbing in the context of a company network. That is not sexy and almost never yields direct results.” But for companies such as Netflix and Zalando, which process enormous numbers of events and petabytes of data daily, it is vital. And (medium-)large Dutch municipalities, with their multitude of domains, show interest too because they struggle with similar problems. In a video Justin Cunningham of Netflix explains how a data mesh simplifies the plumbing.

From push to pull

A paradigm shift is needed. The traditional push model, in which large amounts of data are pushed through ETL pipelines, must give way to a pull model in which business domains deliver data products just like consumer products. “Data wrapped in a package: not a semi-finished product, but an integral product with a nice bow around it.”

The four principles of a data mesh

1. Domain ownership. Instead of letting domain data flow to a central data lake or platform, domains host their own data sets and offer them in an easily consumable way. Ownership of transactional and analytical data is decentralized to the domains closest to the data.
2. Data as a product. Treat the data from the domains as a product and the consumers of that data as customers. Customers, such as data analysts and data scientists, may make demands on quality, accessibility, findability, interoperability, security, comprehensibility, reliability and integrity.
3. A self-service data platform. Create a platform that enables domain teams to share data and removes friction in the whole route from source to use. Its services manage the complete life cycle of all individual data products.
4. Federated governance. Set up an operating model for data management based on a federated decision-making and accountability structure, with a team of domain representatives, controllers and experts in legal affairs, compliance and security. It balances the autonomy of domains with the interoperability of the whole network, so that no single domain can obstruct by refusing to share its data.

Is the data mesh the death blow for the central BI department?

Daan van Beek: “When I let these four principles sink in, I see on the one hand the potential of this new philosophy and on the other the enormous impact on the organization and the enterprise architecture. This could well be the death blow for the central BI team as we know it. Consistently applying the principles in effect means dismantling your central BI department. You will also incur extra costs initially: every business domain must have a powerful data warehouse and servers. And the carefully built-up, centrally bundled BI knowledge is partly lost. A data mesh is a difficult but passable path. Zalando also set up a central data warehouse at first. Do not throw away old shoes before you have new ones.”

Central data warehouse versus data mesh

The main differences between the two architectures.

The snags

There are quite a few snags. Each domain gets its own (small) data warehouse, with all the consequences. In addition you still need a central BI architecture for cross-domain analytics, either through a central data warehouse or with data virtualization, because meaningful analyses usually require combined data from several domains, such as productivity per employee. Van Beek: “BI, wherever you organize it, is technically complex and involves real engineering work. But for large or complex organizations there is a limit to the capacities of a central model, and a data mesh architecture can then reduce the complexity. In essence we do not necessarily need a data mesh, a data warehouse or a data lake, but BI teams that move very close to the business and develop and use business intelligence together with it.”

Five arguments for a data mesh

1. It is based on sound sociotechnical principles. Sociotechnics focuses on the interactions between people and technical solutions in complex organizations. With principles such as self-management, a focus on added value, teamwork, horizontal coordination and decision-making based on partnership and dialogue, it is on the same wavelength as a data mesh.
2. It finally gives data the status it deserves. According to McKinsey there is only one way to get as much value as possible from data: manage it like a product. Every data product should have a product manager and a team, with standards, best practices and quality assurance.
3. Business domains are far better able to judge data quality. The emphasis used to be on centrally certified “golden data sets”. In a data mesh a domain data set only becomes a data product after it has gone through quality assurance locally. The owners of domain data know the details of the operations that produce the data.
4. It paves the way for data virtualization. Data virtualization integrates data from several sources without physically replicating it. The user addresses one virtual layer and always uses the most current source data. The municipality of Sittard-Geleen studied its advantages.
5. Data mesh and Common Ground complement each other perfectly. Large municipalities struggle with huge amounts of data per domain in endless administrations and are expected to keep hundreds, in an extreme case 1,200, applications running. One of the starting points of Common Ground is that data is stored only once and requested at the source. Some municipalities have thus already anticipated a data mesh.

Consequences for the enterprise architecture

Applying the principles of the data mesh in combination with data virtualization has consequences for the ideal architecture. Databases and BI knowledge are concentrated per business domain, close to the source. In fact new, small silos arise. But unlike the old data silos, in which inconsistent, fragmented and often duplicated data became ever more isolated, in the data mesh the information is accessible to everyone in the form of data as a product. The data virtualization layer provides the enterprise BI. Van der Linden: “If you take data quality seriously, you will have to set up an architecture that does justice to the interests of all stakeholders and at the same time lets rusted data flow again.”

Conclusion

A data mesh responds to many pain points organizations struggle with today and can accelerate the ambition to become more data-driven. It is technology-neutral and offers a common language. It is an interesting alternative to the data lakes and central data warehouses that organizations have set up on a large scale. But realize that with a data mesh you turn everything upside down and have to throw the architecture choices of the past overboard. Whether it is an analytical database, a data warehouse, a data lake, a data hub, a data fabric or a data mesh: make sure the choice is well considered.

Further reading

• Data warehouse
• Data quality
• Blog: Common Ground puts BI principles under pressure
• Blog: 6 reasons not to build a data lake

Sources

• Zhamak Dehghani, Data Mesh Principles and Logical Architecture (2020)
• McKinsey, How to unlock the full value of data? Manage it like a product (2022)

About the author

Rick van der Linden studied Industrial Engineering and Management at the University of Twente and has more than twenty years of experience in banking and insurance, healthcare, government and IT. He has worked on almost every aspect of business intelligence and analytics, as a product owner, solution architect and BI consultant.

About Diamond Decisions

Diamond Decisions helps organizations become better by enabling better decisions. We don’t start with data, dashboards or AI, but with the decision that needs to improve, and work back from there to KPIs, data and technology. We do that in four steps: inspiration and alignment, business modeling, analytics and data platform and data-driven decision makers. That way BI and AI don’t remain an IT project, but deliver value every day.