Data technology has finally gone mainstream. Top executives from all around the world largely recognize the critical role of big data in enabling decision automation and digital transformation.
According to the Big Data Executive Survey 2017 from NewVantage Partners, 97.2% of Fortune 1000 companies are already investing in Big Data and AI initiatives as they seek to become nimble, data-driven businesses, and 79.4% of the Fortune 1000 executives fears disruption from data-driven competitors.
Besides, it’s not just large corporations who have been embracing data analytics. As storage capabilities and computing power keep expanding, more and more small and medium-sized businesses are taking advantage of big data technologies.
If you are a business owner or C-level executive and you want your company to stay relevant in the coming years, you must stay up-to-date on the latest trends in the world of big data. Independently from the size of your business, it's time to become familiar with concepts such as cloud computing, real-time analytics and data lakes, and keep track of the continuous evolution that characterizes data technologies.
In this article, we present the 5 main data technologies that went mainstream last year and we analyze the main consequences of the diffusion of these new tools and technologies.
5 Data Technologies That Went Mainstream
1. Cloud solutions
Every year, we exponentially generate more data. The Internet of Things (IoT) is increasing even more the rate at which companies produce, collect and analyze data. As such, companies need to embrace a perfect scalable solution for managing such huge volumes of Data.
IoT clouds are emerging as the main solution to this issue. Not only they are very flexible and adaptable, they also facilitate data integration and guarantee security and reliability.
It is therefore not surprising that the cloud industry is rapidly growing and more and the market now offers more cloud products and services than ever before. We expect the number of cloud vendors and solutions to even increase in the coming years.
2. Dark Data
Dark Data can be defined as the information that organizations generate during regular business activities, but which they generally fail to use for business analysis. The vast majority of data can be included in this category. According to McKinsey and Company, in 2017, only around 1% of corporate Data had been analyzed.
CRM, ERP, SCADA, HTTP and WIFI systems, all generate dark data. Once that information is captured, it can be analyzed to drive new revenues or reduce costs. For example:
- Server log files can be analysed to understand website visitor behaviour;
- Customer call detail records can indicate consumer sentiment;
- Mobile geolocation data can reveal traffic patterns to aid in business planning.
Dark Data remains complex to analyse and often stored in locations where analysis is difficult. However, some highly innovative companies already offer some solutions that simplify dark data capture, preparation and exploration and no one actually knows the limits of dark data.
3. NO-SQL Databases
When most of the data owned by an organization used to be structured, RDBMS systems would fit the needs of most companies. It is not like that anymore.
Nowadays, integrating data from many different sources, such as the corporate website, social media, IoT, sensors and more, means being able to extract a larger amount of better quality insights. Businesses therefore have to manage and analyze data from a large number of sources. This completely revolutionize how we approach and choose database solutions.
As such, NO-SQL databases, also known as non-relational databases, are disrupting the world of databases. If you work with data, you’ve probably already heard of products like MongoDB, CouchDB, Neo4j, Redis or Apache Cassandra.
Despite being born in the 70’s, these solutions use a dynamic schema that allows firms to store and retrieve unstructured data. They are therefore a perfect ft to answer the needs of modern corporations. While SQL databases are still the preferred solution for most businesses, NO-SQL can be expected to soon become the most diffused option on the market.
4. Streaming Analytics
The huge improvements seen in data technology in the last few years have enabled the rise of real-time analytics solutions. New technologies are making data-driven real-time decision-making much more affordable.
Like in the case of dark data, streaming analytics offers insights that can’t be found anywhere else. Further, as most companies understand how to store and process big data, how rapidly they are able to analyze their data has now become the main source of competitive advantage.
Some of the main benefits of streaming analytics include a better comprehension of each market and industry the company is working in, enhanced customer profiling capabilities, improved business process management and demand sensing. Unsurprisingly, real-time decision-making has rapidly become a must-have for data-driven companies.
5. Artificial Intelligence
The spread of automated data technologies and the gradual combination of data engineering, data science to data analytics, are making AI mainstream across many industries. As computers become more powerful, more and more companies have the means to build artificial neural networks and explore the potential use cases of machine learning (ML) and deep learning (DL).
As such, many organizations have been embracing ML and DL in 2019 to answer specific business needs and drive growth. Neural nets and AI are already being used for image recognition, text translation, voice recognition, game-playing and biology research, just to cite a few examples.
While a generalised platform that promotes the democratization of AI still doesn’t exist, many companies show signs of being ready to embrace AI, ML and DL. We expect this technology to rapidly go mainstream as businesses discover and develop new applications.
Conclusion: Skills Shift as Data Technology Spreads
Data technology is going mainstream and this is bringing many advantages in our lives. Unsurprisingly, the demand for big data experts has never been so high. However, as technology evolves, in-demand skills shift.
Until recently, most companies had their data stored and organized in a random and siloed manner, which made it almost impossible to extract meaningful insights from all that information. For most businesses, integrating the huge amounts of data that they had been collecting and storing for years was the only priority.
Now that more and more companies have implemented some standards for data collection and storage, the focus is shifting towards data exploitation. Indeed, standardization fosters data democratization and the diffusion of end-user focused self-service analytics and visualization tools is lowering the day-to-day operational skills needed to successfully run a data analytics operation.
As such, rather than expert statisticians and programmers, companies look more and more for data-savvy employees with a good understanding of the business and industry they are contributing to. Most businesses working with big data technologies are now looking for data analysts, also known as “citizen data scientists,” who are usually only required bachelor-level education.
Nonetheless, this is normal when technologies go mainstream. It doesn’t mean there will be less opportunities for experienced data professionals.
Engineers who are able to work with the core tools (databases, Spark, Airflow, etc.), as well as neural networks and machine learning experts, can expect to be in high-demand in the coming years. In addition, some companies are also opening new high-profile positions, e.g. Chief Data Officer, that will fit more experienced candidates and which we expect to spread quickly in the corporate world in the near future.
In conclusion, data technologies are going mainstream and this is good news, as these tools can promote positive change in our society. However, when technology gets better, while new opportunities arise, new challenges also emerge.
Keeping a close eye on the trends in the data industry will help you grab the advantages and overcome the difficulties. And even if you are still not familiar with what is going on in the data industry, there is no need to worry: you are still in time to get on track.
