Data Integration: Where Are We Heading? Key Takeaways from the Gartner Magic Quadrant of Data Integration Tools 2018

Data has become an integral part of today's business operations. And as the number of data users is growing across all industries so are the demands for agile, easy-to-use data integration solutions. This development is clearly reflected in this year’s Gartner Magic Quadrant for data integration tools.

The MQ is an annual report designed to give its readers a wide-angle view of key trends and evolutions that are shaping the data management market. Below, we reflect upon some of the most important takeaways and their impact from Data Virtuality’s perspective.

Data Virtualization: A Rising Star with a Catch

In its report, Gartner notes that “[i]n 2018, traditional integration is beginning to shift from the bulk/batch dominance for delivery in the market, but only slightly.” A truly exciting assessment!

What does that mean?

Traditionally, ETL technologies have dominated the data integration market with 80% of end-user organizations still making use of bulk/batch processing. However, data virtualization is making inroads into the ETL territory at every turn. According to Gartner, more than 40% of businesses already utilize data virtualization technology, and the number is growing steadily. This development is also reflected in this year’s Gartner’s Hype Cycle for Data Management. The analysts forecast that data virtualization is less than two years away from mainstream adoption, or the “Plateau of Productivity”, as Gartner has branded it.

But, alas, it’s not all roses and sunshine!

As data virtualization is becoming more widely used there are also more vendors that specialize in this technology. However, Gartner repeatedly points to the disadvantage they all have in common: data virtualization solutions alone cannot address the full scope of data integration workloads. Thus, users need additional relevant, i. e. competing, tools to complement their data integration systems.

We think that hybrid integration solutions, especially the Logical Data Warehouse, are the key to overcoming this challenge. The Logical Data Warehouse is a single source of data truth platform that combines ETL and data virtualization into an ecosystem of multiple, fit-for-purpose repositories, technologies, and tools that interact synergistically and provide performant enterprise analytical capabilities. And that’s only the beginning!

Cloud Integration in Hybrid and Multi-Cloud Scenarios: A Modern Classic

Another major shift that Gartner identifies in the MQ is “customers asking for hybrid deployment (cloud and on-premises), as in 2017, but now with the expectation of multi-cloud and cloud-to-cloud integration.”

Considering the current data environment, this statement is hardly surprising. More and more businesses have their data distributed among different storage facilities – on-prem as well as in the cloud. It’s a big challenge to effectively manage the scattered data that requires a unified, fit for purpose data integration process ...

Sounds like a classic data integration problem? Not quite!

Traditional ETL-based data warehousing approaches are not agile or responsive enough to meet the challenges of these scenarios. Similarly, data virtualization technologies, while clearly being a significant step in the right direction, also reveal serious drawbacks, such as limited scalability and a heavy reliance on the speed and stability of the source systems.

This is where the Logical Data Warehouse comes in:

The Logical Data Warehouse is a key technology to solving the limitations of its predecessors by combining data virtualization and an automated ETL engine with a common query language (SQL). No matter where the data is stored or in which format. The Logical Data Warehouse is specifically designed to instantly query, manage and integrate all types of data from almost any database and cloud service. The solution can be deployed on-prem and in hybrid as well as multi-cloud scenarios.

Self-Service Data Integration: A Risky Business

The MQ also highlights that “[d]ata integration is everywhere and is everyone’s responsibility”. In fact, Gartner reckons that “[r]ole-based development and deployment management is now incumbent on the data integration to deliver.” Simultaneously, however, the report challenges “the belief that anyone can integrate” and asks to consider instead “if everyone should integrate”.

But let’s start from the beginning:

Traditionally, most data integration systems, though very well structured and curated, have had one major problem: they are extremely slow and require expert handling. This has left business users, who expect speedy and comprehensive analytical data access, high and dry.

Enter self-service tools!

In response to this dilemma, self-service tools were introduced to the market. These tools enable business users (whom Gartner calls “citizen integrators”) to access and work with corporate information without expert support. How? By ‘blending’ or locally integrating data from the data warehouse with any other data sources not stored in the data warehouse. In short, self-service tools are indeed flexible, relatively easy to implement and enable data integration to a certain degree. But they also have clear disadvantages. The most prominent being their lack of governance and manageability which results in chaotic reporting practices, contradictory results, and competing data stores (to name just a few).

So, we need an in-between solution that combines the flexibility of self-service tools with the strong data governance capabilities of the more sophisticated systems: the Logical Data Warehouse. In a nutshell, the Data Virtuality Logical Data Warehouse bridges the gap between technological complexity and self-service initiatives by permanently optimizing the backend data structure without influencing the business view. Thus, it simplifies and accelerates the access to the information final users and citizen integrators require and exposes the data in a business-friendly manner.

Metadata: A New “Boom”

Last but not least, the era of metadata has begun. Or, as Gartner puts it: “The biggest change in the market in 2018 is the shift from an anticipated future demand for metadata-driven solutions to a current market expectation that these solutions will be delivered as part of the data integration platform. There is currently no forgiveness for inadequate metadata capabilities.”

Metadata management is increasingly becoming the heart of data integration capabilities and this is also reflected in the development of Data Virtuality’s integration solution. Our Logical Data Warehouse comes fully equipped with advanced capabilities in metadata management. Among those are a metadata catalog, that enables elaborate metadata search, optimized metadata creation features and a web surface for business users – to name just a few.

So, where are we now?

To sum it up, at Data Virtuality we think the latest edition of the Gartner Magic Quadrant for Data Integration corroborates our market hypotheses: a shift to highly distributed data architectures coupled with business-centric data management expectations and elaborate metadata management capabilities. Yet Gartner repeatedly emphasizes that neither traditional data warehousing approaches nor data virtualization technologies alone can cope with these demanding developments. This confirms our belief that data consumers need both data virtualization as well as bulk/batch technologies to successfully realize their data integration projects. This idea of a hybrid integration solution is realized in the Logical Data Warehouse, an approach whose role is going to expand significantly in the coming years. In Gartner’s words: “the influence of hybrid platforms for integration is rapidly gaining ground.” And the Logical Data Warehouse is at the very front of this development.

Gartner Disclaimer

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.