Big Data London 2023 was a huge event, and the buzz this year seemed to be centralized on data mesh, not necessarily on its successes but on how data culture, lack of buy-in, and deficient data fluency impact its success.
I sought out fellow practitioners who had been through the school of hard knocks with at least one data mesh implementation. The common thread to what works is the data contract and treating data assets as products. What doesn't work? The inability to gain buy-in across the company despite executive sponsorship.
We've all read the articles, blog posts, and books about how data mesh is the next best thing since sliced bread. Yet, many are struggling or have given up entirely on the idea—relabeling it as a “data mess.”
Big Data London 2023 was a huge event, and the buzz this year seemed to be centralized on data mesh, not necessarily on its successes but on how data culture, lack of buy-in, and deficient data fluency impact its success.
I sought out fellow practitioners who had been through the school of hard knocks with at least one data mesh implementation. The common thread to what works is the data contract and treating data assets as products. What doesn't work? The inability to gain buy-in across the company despite executive sponsorship.
We've all read the articles, blog posts, and books about how data mesh is the next best thing since sliced bread. Yet, many are struggling or have given up entirely on the idea—relabeling it as a “data mess.”
Let's set the background. Back in my days in the fintech industry, I led the company's first productized deployment of data mesh.
As head of intelligence automation, I was engaged by our Chief Risk Officer to create a solution that would bring together data sets across siloed business teams — creating a robust view of the profitability of customers. He didn't care about the technical solution. The priority was the outcome: Trusted, curated data for his analytics team to leverage.
The promise of quality, business context, and governed data use was also extremely appealing. We had the support of one of the world's leading technology groups and operated using best-in-class centralized engineering tools. We were also free and autonomous to experiment with the solution to deliver on his request. We chose to take on a data mesh build.
To this day, I'm so damn proud of that team. We accomplished so much:
The risk and finance organizations leaned in and supported the data mesh approach. Our data governance partners loved the concept, especially the computational federated governance and the opportunity to integrate with the centralized MDM system.
Most importantly, however, we solved the business problem. Analysts who used our solutions loved the self-service, intuitive feedback loops, and new visibility into data quality and SLAs from an observability standpoint.
Yet our project was abandoned during a reorganization.
As I wandered the event in London last week, I sought some answers from practitioners who had taken the leap to build an actual data mesh solution. I asked the big question: Why do so many data mesh implementations fail? As somewhat expected, here were the responses:
Looks like the fintech I worked with wasn't the only one to start, stop, and get lost on data mesh.
Industry experts on this sociotechnical revolution are playing musical chairs right now. They are highly sought after to lead the transformation but then quickly abandoned if, in mere months, they do not deliver upon the promised value of computational federated governance, experiencing and leveraging data as a product to gain ROI, self-service with business context for their analysts, and new levels of ownership in business domains.
In my team's case, on the technical side of adoption, our centralized data engineering team was not easy to convince nor supportive of the initiative. They had their objections:
On the business side of adoption, other teams outside of ours had a variety of solutions and BI tools, rogue data sets, and differing processes for compiling analytic data. The resistance to change was apparent, though slightly different from the engineers:
Ironically, analysts and scientists are the ones complaining that 80% of their time was spent finding and cleaning data for analysis.
Recently, I was asked if one could sell data mesh (and yes, I know there are companies out there selling it). And my answer is no — at least, not yet. Not without first addressing both the cultural and change management aspects, along with the technical features as we implement. If we don't implement all four principles simultaneously, we will continue to see failures.
Consider this. There are some fantastic tools and capabilities in progress that manage components of the data mesh concept. However, the vast revolution in a culture that is required to make data mesh stick is still quite a distance away for many, if not most, organizations.
That said, the companies that are succeeding with data mesh will have a substantial competitive advantage:
Enterprises that can adapt to cultural shifts, address organizational change, and process engineering, not just the data engineering aspects of this solution, will become tomorrow's market leaders.
To create true and lasting change, we must revolutionize how our companies think about data. While every company will be different in where it starts, the core foundation still comes down to embracing a data-driven mindset and ensuring every piece of the organization has accountability to ensure it is adopted. It's all about culture, adoption, and process — perhaps it's time to shift our conversation there.
I want to hear from you — have you been a part of a successful data mesh implementation that scaled? Where are you on your journey? How did you address the cultural hurdles to success?
