In the testing and development of connected and automated vehicles, petabytes of data is produced and collected, which must be protected and managed. As a result, in the near future, we will witness a data surge unlike anything the automotive industry has ever experienced.
Essentially, autonomous vehicles are moving computers that continuously process and produce data. Each autonomous test vehicle generates between 5TB to 20TB of data per day, all of which must be received, stored, protected, analyzed – and retained for research and legal purposes.
It’s not only the vehicles themselves that are responsible for data growth – the entire ecosystem of suppliers, service providers and partners is also more data driven. Not only will these organizations have to build adequate IT infrastructures to efficiently maintain and manage critical data, they will also have to deploy technology to enable the flexible exchange of data between multiple parties. The data itself requires intelligent interaction at all points – from the onboard computer to the data center and the cloud.
OEMs have highly complex IT infrastructures that often consist of siloed structures, which hinder efficient data management and impact data security. The key to intelligent data protection is the classic ‘3-2-1’ rule, where at least three copies of data are stored on two different media, while one copy is held off-site. New concerns around ransomware suggest that one of those copies should also be stored off-line. Even in the modern mobile and multi-cloud era, the ‘3-2-1’ rule is still the best way to protect data against hardware failures, cyberattacks, data corruption and data loss.
In the analysis and simulation of artificial intelligence (AI) solutions, autonomous vehicle manufacturers and suppliers use high-performance storage facilities. For longer-term data archiving and retention, such high-performance storage isn’t required. Where low cost and high security are the primary concern, tape remains the best media option. Public cloud solutions are also viable for storage that requires a high degree of flexibility.
IT teams should always follow tiered storage principles when choosing their storage medium. Tiered storage approaches move data to optimal storage media according to different requirements such as access time, security, performance and cost – with the goal being to reduce overall IT costs depending on the business need.
Automated tiering
In order to work effectively with large data volumes, storage tiering must automatically match data to its optimal storage media. This ensures the organization’s accessibility, performance and cost goals are met – all without impacting the user experience. These automated solutions – commonly referred to as active archives – enable end-users to locate and launch their files without knowing where it physically resides, which is critical in connected car and autonomous vehicle development environments that rely on collaboration. As long as data is always placed on the most suitable media, automotive businesses will be able to manage both their productivity and their costs.
Ultimately, car manufacturers and technology providers have to recognize the importance of large-scale, intelligent data management in the mass production of AVs. There’s no need to reinvent the wheel here – the right data storage technologies already exist. Auto makers and their suppliers must prioritize these cost-effective data management solutions, in order to further their product development in the years to come.
Anthony Tocco has a wealth of knowledge in ADAS projects, including how lidar, radar, video, GPS and many other IoT sensors create data. His 24 years of experience with technical workflows, high-performance storage environments, and disaster recovery/business continuity has helped him with ADAS solutions in HIL, SIL, PIL, and MIL environments.