Data Mesh is a socio-technical data management paradigm proposed by Zhamak Dehghani. It challenges the traditional monolithic data architectures that were rather technology-focused with a business-centric decentralizing approach to ultimately make the data usable in companies. Data Mesh confronts the current enterprise data architecture to:
make a large number of sources fully manageable
enable a diverse set of consumers
Zhamak Dehghani who coined Data Mesh defines this concept as follows: “A data mesh considers domains as a first-class concern, applies platform thinking to create self-serve data infrastructure, treats data as a product, and introduces a federated and computational model of data governance.“
Why Data Mesh Evolved
The efforts around Data Mesh aren’t new. Like the concepts Data Fabric, Logical Data Warehouse, Data Lakehouse, etc, the ultimate goal of Data Mesh is to make the data accessible and usable for all users, especially the business side.
So what makes Data Mesh different?
Mindset shift of data to be an asset that is treasured to a product that is used
Big drivers of Data Mesh are communities. Communities helped to be effective in creating new data and partner ecosystems and spread the word about this new concept. The enablements of data sharing, insight sharing, co-development of models, and data-driven products as well as services are all elements that laid the foundation of Data Mesh.
What creates quite a confusion about this concept is the fact that Data Mesh drifts away from technology. It is often misunderstood that technology can be neglected. But to enable Data Mesh, discussions about technology cannot be ignored. Only with the right data integration, management, governance,… tools, a user-centric approach can be provided.
Watch this video to learn more:
How Data Mesh Works
To enable the paradigm shift of moving beyond the traditional monolithic approaches, the data mesh is based on four key pillars.
Domain-oriented decentralized data ownership and architecture
Decentralization and distribution of responsibility to the domains
Data should reside and be managed by the people who are closest to it as they are most familiar with it
Users can then make the adjustments in the most efficient way.
Data as a product
Perspective needs to change from data being an asset to data being a product that can be discovered, understood, and trusted
Data consumers should be treated as its customers
Domain owners need to provide data that is of high quality, trustworthy, and short time-to-market to satisfy the customers
Self-Service data infrastructure as a platform
Platform needs to enable domains in a self-service manner
Capabilities for data integration and transformation, implementation of security policies, data lineage, and identity management
Federated computational governance
Governance model with global standardization and harmonization is required
Standardization efforts can address concerns in regards to data semantic/syntax modeling, metadata formatting, identification management, etc.
Data Mesh vs Data Fabric
“Data Mesh is the approach, Data Fabric is the platform.” Unlike the public voices that imply contradiction between data fabric and data mesh, these two concepts actually lead in the same direction.
Data Mesh is more philosophically and organizationally oriented, asking “What’s in it for me?”
Data Fabric is rather driven by the technical side, reflecting on the questions “How do I build this and which components do I need?”
Companies often start by deploying a data fabric and get some first value from that. With the priority on the flow and pathways of data, the elements of data integration, schema governance, and types of pipeline capabilities (ETL/ELT, data virtualization, microservices, APIs, streaming, etc.) in the context of SLAs, etc. is defined in Data Fabric. In the next step, the business outcomes, and time to value are added with Data Mesh.
Watch this video to learn more about the full evolution of Data Mesh:
“A data fabric is a technology-enabled implementation capable of many outputs, only one of which is data products. A data mesh is a solution architecture for the specific goal of building business-focused data products.”
If you want to better understand the differences and similarities of Data Mesh and Data Fabric, watch this video:
Main Advantages of Data Mesh
The Data Mesh concept differs from the other ones as it is technology-agnostic and focuses more on the human/socio part of the data management challenges. The data mesh framework claims that the previous concepts concentrated too much on the technology and thereby missed to fully understand and address the needs of the business that ultimately uses the data for insights. With the decentralized approach, the data mesh tries to bridge the gap between the business needs and the technology. On the technical side, it recognizes and respects the distributed nature and topology of the data and the different use cases that it can enable. On the human side, it looks at the individual personas of data consumers, their diverse access patterns, and their domain-specific knowledge.
Pillar
Pro
Con
Domain-oriented decentralized data ownership and architecture
Removes the bottleneck of centralized infrastructures: a separate entity that takes care of all tasks related to data management isn’t needed, e.g., data scientists looking for data in a data lake environment.
A decentralized approach can lead to an increasing number of data silos and suffering data quality and a decrease in data quality => data needs to be treated differently.
Data as a product
This is mainly a philosophical shift, so no technology has to be bought for this.
Changing people’s approach to data takes time.
Self-service data infrastructure as a platform
Business becomes less dependent on IT and, therefore, is more agile.
Managing data infrastructures is complex and requires special skills which won’t exist in all domains. In order to still enable the different domains.
Federated computational governance
Ensure that rules and regulations are adhered to, and the company is compliant.
Difficult to ensure a healthy and interoperable ecosystem in this decentralized set up.
This framework surely has a lot of potentials. However, it is still in an early emerging stage with very little real-life implementations. Many weaknesses and challenges are still unknown and unclear. It will be interesting to see how it evolves in the next few years!
Use Cases enabled by Data Mesh
All data-related use cases can benefit. But when you take a closer look, use cases with analytical and transactional systems in which digital systems are embedded with intelligence are most favorable to Data Mesh. Below, you can find an exemplary list of use cases:
Customer 360
More automated processes providing better personalized and contextualized customer experience. Results are reduced average handling time, increased first contact resolution, and improved customer satisfaction.
Marketing
Marketing teams are enabled to run the targeted campaigns to the right customer, at the right time, and via the right channels.
Data Privacy
Customer data can be protected by complying with the ever-emerging regional data privacy laws, like GDPR. Security rules can be easily applied through the integration of external data governance, policy, and security tools (such as Collibra) on the global level prior to making it available to data consumers in the business domains.
IoT Device Monitoring
Insights into the device usage patterns help to continually improve product adoption and profitability.
AI and Machine Learning Training
Machine learning (ML) and artificial intelligence (AI) models can be easily fed with data from different sources to help them learn, without running the data through a central place.
IT and DevOps
Data latency can be reduced by providing instant access to query data from proximate geographies without access limitations.
Loss Prevention
The domain-oriented decentralization allows to analyze the data and run fraudulent behaviour models on a local level and thereby, detect and prevent fraud in real-time.
M&A
The combination of decentralized data ownership and federated governance allows data sovereignty and data residency on a regional level while complying with data governance rules on the global level.
Interested in seeing how you can get started?
Book a demo and learn how you can enable Data Mesh with the Data Virtuality Platform:
While caching offers certain advantages, it's not a one-size-fits-all solution. To comprehensively meet business requirements, combining data virtualization with replication is key.
While caching offers certain advantages, it’s not a one-size-fits-all solution. To comprehensively meet business requirements, combining data virtualization with replication is key.
On-Demand Webinar: Data Architectures for Data Science Using Data Virtualization
