BIG DATA: Great opportunities and challenges

Big Data on a daily basis

In 2009, the NBA began using a video system to track players and ball movement. Today, in 2021, each NBA franchise has a team of data analysts who analyse how competitors play and provide valuable and non-intuitive knowledge to coaches.

Does Luka Doncic shoot more from the left side? Is it cost-effective to facilitate James Harden’s drive to the basket so that he doesn’t shoot so many three-pointers?

This is just one of the multiple applications of Big Data, a discipline that requires new specialists and specific techniques. When we talk about Big Data, we cannot think of spreadsheets nor conventional software applications.

For instance, it is estimated that an autonomous car can generate 40 terabytes of data in 8 hours of driving, which is much more than what a person generates today. The key factors, the 5 Vs, that define Big Data are Data Volume, Velocity and Variety, but also Veracity and Value.

But why do we have so much data now? Wasn’t there data before? In 2001, I was a Six Sigma Black Belt at Alstom, a train manufacturer, and I can say that one of our biggest challenges was finding data and the quality of that data.

But now SCADA systems, the Internet of Things, the generalisation of imaging systems (thermography, video and photo, medical imaging, radar images, satellite images…), billions of mobile phones… We have a huge amount of data at our disposal.

Anyway, having the data would not have been enough for the discipline of Big Data to emerge with such force. We have mentioned the Internet of Things, or mobile phones, but we can also point to cloud technologies and Artificial Intelligence, and very soon 5G. The technological revolution we are living through arises because these technologies feed off each other.

Horizon Europe logically recognises the importance of Big Data, and for this reason we can find opportunities for developments in “Big Data” both in its applications (Health, Environment, Energy, Mobility, Advanced Manufacturing, Security, etc.) and in the specific development of new techniques (whether in data mining, data collection and storage, data cleaning, data analysis and consumption) and in the impact of Big Data on our society and economy.

In Horizon Europe we find 47 topics for 2021 and 2022 related to Big Data. Some of them are:

5 essential tips

At ZABALA, our advice for projects involving Big Data are:

• Rely on Big Data specialists: data scientists, software and cloud experts, etc.
• Seek interdisciplinary knowledge and encourage relationships and work between people from different disciplines.
• Take into account the ethical, social and economic aspects of the data from the outset.
• Likewise, take into account the security of the data from the beginning.
• And finally, think about whether your data can offer value to other applications, whether it should be open data, restricted data, closed data, etc.