A new dataset promises to accelerate the development of autonomous vehicle (AV) technology by providing researchers with a wealth of previously unavailable real-world driving data captured from multiple vehicles over repeated trips.
THE MARS (MultiAgent, multitraveRSal and multimodal) dataset, presented by researchers at the NYU Tandon School of Engineering in collaboration with an autonomous vehicle company May Mobilityoffers a unique combination of features that sets it apart from previous efforts in the field.
The NYU Tandon team presented a paper on MARS last month at the IEEE/CVF Computer Vision and Pattern Recognition (CVPR) conference, the premier annual computer vision event. The MARS dataset is Publicly available.
“Datasets for autonomous vehicle research typically come from a single pass of a vehicle at a certain location. MARS offers much more possibilities because it captures real-world interactions between multiple autonomous vehicles traveling on fixed routes hundreds of times, including repeated passes at the same locations under varying conditions,” said Chen Fengthe project’s principal investigator. Feng is an assistant professor at NYU Tandon and works on computer vision for autonomous vehicles and mobile robots, including NSF CAREER Award funded this project.
The dataset – curated by Feng’s Automation and Intelligence Laboratory for Civil Engineering (AI4CE) and May Mobility engineers – were collected using a May Mobility fleet of four autonomous Toyota Sienna Autono-MaaS vehicles operating within a 20-kilometer zone encompassing residential, commercial and university areas in a U.S. city.
May Mobility’s FleetAPI subscription service provides access to real-time and historical data from its vehicles. This allows data partners like NYU Tandon to access real-world information including sensor data (LiDAR, camera), GPS/IMU, vehicle health, and more.
“The MARS dataset allows us to study both how multiple vehicles can collaboratively perceive their environment with greater accuracy and how vehicles can gain a detailed understanding of their environment over time,” Feng said. “We could not have assembled it without the unprecedented access May Mobility provided us to its large-scale, real-world data. The result is an important step toward ensuring the safety and reliability of autonomous vehicles on our roads. Additionally, this collaboration sets a precedent for industry-academia partnerships that benefit the entire field.”
“We believe that transparency and data sharing can do more than help our customers, it can help the next generation of innovators push the boundaries and come up with their own big ideas,” said Dr. Edwin Olson, CEO and co-founder of May Mobility. “As we continue to build bridges with academia, their research will pave the way for more innovation at May Mobility and across the autonomous vehicle industry.”
NYU Tandon began planning with May Mobility in November 2022. Since then, NYU Tandon researchers have worked closely with May Mobility engineering teams to access daily operational sensor data from the studied fleet group and selected more than 1.4 million frames of synchronized sensor data. This included scenarios in which multiple vehicles encountered each other on the road, providing valuable insights into how autonomous vehicles might cooperate and communicate in the future.
One of the most significant aspects of MARS is its “multi-passage” nature. May Mobility engineers and NYU Tandon researchers identified 67 specific locations along the route and collected data on thousands of passes through these areas at different times of day and in varying weather conditions.
“This repeated observation of the same locations is crucial for developing more robust perception and mapping algorithms,” said Yiming Li, first author of the paper and a doctoral student in Feng’s lab. recently won the prestigious NVIDIA Graduate Fellowship“This allows us to study how autonomous vehicles could use prior knowledge to improve their real-time understanding of the world around them.”
The release of MARS comes at a time when the autonomous vehicle industry is striving to move beyond controlled test environments and navigate the complexities of real-world driving.
Because the dataset is collected from multiple commercial vehicles in real-world use—not from vehicles deployed expressly for data collection, single autonomous vehicles, or data simulations—it can play a particularly vital role in training and validating the artificial intelligence systems that power autonomous vehicles.
To demonstrate the potential of the dataset, the NYU Tandon team conducted initial experiments in visual location recognition and 3D scene reconstruction. These tasks are fundamental to an autonomous vehicle’s ability to localize itself and understand its environment.
“MARS is a powerful example of industry-academia collaboration at its best. Collecting data from our real-world operations opens new avenues for autonomous driving research in collaborative perception, unsupervised learning, and high-fidelity simulations,” said Dr. Fiona Hua, Director of Autonomy Perception at May Mobility. “We are only scratching the surface of what is possible with this dataset and look forward to the possibilities that will unfold as we work hand-in-hand with researchers to solve current and future autonomous driving challenges.”
The collaboration and release of the MARS dataset builds on NYU Tandon’s broader efforts to create safer mobility and improve the accuracy and efficiency of autonomous driving algorithms, a commitment that May Mobility shares.
Feng before worked on a project which compiled a dataset of more than 200,000 outdoor images to test a range of visual place recognition (VPR) technologies to improve navigation in complex urban environments.
Earlier this year, NYU Tandon has been selected for the National Artificial Intelligence Research Resource (NAIRR) pilot project of the National Science Foundation and the U.S. Department of Energy, along with another project advancing autonomous vehicle research. Led by Associate Professor Chinmay Hegde, NYU Tandon’s NAIRR project will develop techniques to safely deploy advanced AI models in autonomous vehicles, focusing on systems that process both visual and textual data.
About May Mobility
/Public dissemination. This content from the original organization/authors may be of a timely nature and edited for clarity, style, and length. Mirage.News takes no institutional position or bias, and all views, positions, and conclusions expressed herein are solely those of the author(s). See the full story here.