Datumize Data Collector (DDC) is a lightweight, high-performance, streaming enterprise data integration software focused on data collection for hidden, complex and disparate data sources, most of the time unexplored due to the incapacity to access them with other technologies.
Datumize has extensive experience in deploying its data capturing technology for ingesting network transaction data, which is ubiquitous in our modern society. Not every byte of data transmitted over a network is stored because the sheer amount makes is impractical in most cases; however, this network traffic hides a massive potential in terms of Dark Data.
Tons of in-transit data remain hidden and unleveraged due to the difficulty of collecting and processing these temporary transactions flowing over the network (B2B API, XML integrations, read-only transactions, product searches, quotes, inventory checks, etc.) without a major upgrade to the backend systems.
Network Sniffing and Deep Packet Inspection (DPI)
Datumize Data Collector uses network sniffing techniques and deep packet inspection (DPI) to collect the network transit data. By doing so, we can tap into ongoing transactions without causing any overhead and without the need of modifying backend systems. The software acts as an observer, ingesting all these temporary data and supporting even "data drifts": the reconstruction of HTTP conversations even with missing network packets or incomplete XML.
The data processing capabilities of Datumize Data Collector also allow assembling IP, TCP, UDP, HTTP, etc into meaningful XML and JSON structures. Once processed and prepared, the data can be transferred into files, databases, Big Data platforms, or event Streaming Analytics platforms (via Kafka Streams).
One of the relevant success cases in where we are using our Datumize Data Collector (DDC) software is in Vueling: this leading Spanish airline is now ingesting all the network transaction data from its integration with a relevant third-party, the online travel agency (OTA) E-dreams, via the deployment of the Datumize Data Collector (DDC).
By collecting and preparing these data, Vueling is now able to feed their BI tool with the information needed to understand the transaction requests and responses, the real demand from this flight distribution channel, and in consequence, better plan their portfolio, routes, and offers.
