Every minute, we are sending over 204 million emails and sharing 2.5 million pieces of content via social networks. By the year 2020, the IoT will comprise more than 30 billion connected devices generating 600 ZB of data per year. The amount of stored and non-stored data is growing every day. To make sense of this data, reveal its patterns, and to be able to use it for operational improvements and more efficient strategic decision making is still a challenge.
Without the structure and clear purpose, data is unusable. It's nothing but tons of unrelated, chaotic information without any insights. One must understand data to create added value. Only when the structure of this data is thoroughly analyzed and understood, with a clear application vector assigned to it data can be transformed into intelligence.
An exponentially growing asset for companies
Data is a valuable resource. Collected and stored in all areas of life and business, the global volumes of data are doubling every two years. Should we succeed in evaluating this data intelligently and comply with the principles of data diversity and data sovereignty, enormous potential for value creation can be generated.
Big players in the IT industry, research institutions, and numerous startups have been somewhat active in the data field. In many sectors, technologies for the economic utilization of the data collected are also very advanced, although, on a large scale, the exploitation of data is still in its infancy. Being the "digital capital" at the heart of digital transformation, data itself becomes a critical capability and pathway to realizing value from it.
Today, every company has large datasets, but only a few of them are indeed able to convert their big data into smart data. Big Data means all data. But data as such is meaningless. For the companies to be able to obtain substantial benefits from this data, they have to turn it into actionable data.
Big Data; the volume
The term Big Data was introduced by O-Reilly Media back in 2005 to refer to "a set of data that is so large it just can't be managed and processed through traditional business intelligence tools." It's so voluminous and complex that it can't be analyzed with the standard data processing applications. Since then, the ever-increasing number of companies learn to deal with massive amounts of information to make better, more informed decisions.
Smart Data; the value
In contrast to Big Data, Smart Data is actionable and does make sense and has a clear purpose. It's not about the volume of the data you are collecting - it's about the actions you take in response to that data. It's a concept that developed along with the development of algorithm-based technologies such as artificial intelligence and machine learning.
Smart Data is usually generated close to the data source, using edge computing technologies. So instead of collecting all data from a source, we process the data in the source to end dumping in our data lake only the valuable data; the Smart Data. They distinguish between two main types of smart data. The first type refers to the data collected by smart sensors and associated with the Internet of Things (IoT) movement. In this case, an intelligent data entry point captures information and uses it for real-time decision making.
Sensors collect external data inputs and intelligently react in fractions of a second. This type of Smart Data is also known as "sensor or fast data" and stands at the base of many modern innovations, such as smart factories and self-driving cars. The other kind of smart data is big data that has been processed and turned into actionable information, empowering the organization that owns it.
Dark Data; the hidden opportunity
Dark Data is not just the small portion of big data; it's the most significant chunk of data with a massive amount of potential that is still ignored by the companies. Dark Data is the data that lies below the surface, hiding within the company's internal networks and holding piles of relevant information that can be moved to the data lake and generate vital business and operational insights.
Dark Data is the non-stored, and unstructured data organizations produce during their regular activities, and according to KPMG, it represents more than 80% of total data. Then, dark data is a great hidden resource that flows untapped through many organizations.
To get the most profit of the dark data, a company should capture, unlock, and transform this data into compelling insights. But first, we need to be able to identify where and what data is still dark in our company, and what's the value we could get from it.
