Neural network-based 'PilotNet'
Recognize different road environments depending on lane markings, road pavement conditions, time of day, and weather.

NVIDIA developed 'PilotNet,' a neural network-based system that learns vehicle driving by observing drivers' behavior. Furthermore, they developed a tool to verify which factors this neural network system prioritizes during decision-making while driving. Utilizing this, they built a self-learning system and made it possible to verify how the vehicle makes decisions.

AI that learns driving by observing human driving styles
NVIDIA is utilizing its proprietary AI vehicle, BB8, for the development and testing of DriveWorks software. To date, NVIDIA has used Ford, Lincoln, and Audi vehicles with BB8 and plans to use vehicles from other manufacturers in the future; therefore, vehicle manufacturer and model information are not key factors in the development of this AI vehicle. The element that enables BB8 to demonstrate the power of deep learning as an AI car lies in the deep neural network that interprets images from the front camera to issue driving commands.

Training was conducted to enable a deep neural network to learn the driver's driving behavior and drive the vehicle autonomously. The network utilized cameras mounted on the vehicle to record what the driver was looking at, and then correlated those images with data from the driver's decision-making during driving. NVIDIA recorded a significant amount of driving data across various environments. In addition to diversifying road environments, such as roads with and without lane markings, unpaved roads, and highways, records were recorded at different times of the day under varying lighting conditions, and a variety of weather conditions were also incorporated.

The trained network autonomously learned how to drive BB8 through observation, without any code-based commands. The trained network became capable of issuing real-time driving commands even in new environments. You can see it in action in the following video.

The thought process of artificial intelligence
As PilotNet went into operation, NVIDIA developed a visualization map to examine its decision-making process, which shows which elements PilotNet prioritizes when observing an image.
Through this visualization map, NVIDIA was able to identify what PilotNet prioritizes most when receiving new information from vehicle cameras, as shown below. The example below shows a visualization overlaid on an image recorded by a vehicle camera, with points where PilotNet places high priority marked in green.


Through these visualizations, NVIDIA was able to confirm that PilotNet focuses on the same elements that drivers focus on, such as lane markings, road edges, and other vehicles. PilotNet learned without direct commands, just as real people learn to drive; it learned through observation which factors are important for decision-making in a driving environment.

“The advantage of using deep neural networks is that vehicles can perceive situations on their own, but it is difficult to achieve substantial progress if developers do not understand how these networks make decisions,” said Muller, Head of Development. “Through this tool developed to verify the network’s decision-making process, NVIDIA can identify the information needed to improve the system. Developers cannot explain to the vehicle exactly what it needs to do, but they can show the driver’s driving behavior so that the vehicle can learn on its own, and the vehicle can now show developers what it has learned.”

If autonomous vehicles enter mass production, various artificial intelligence neural networks and diverse technologies will be utilized for vehicle operation. In addition to the pilot network that controls driving, various networks trained to focus on specific tasks such as pedestrian detection, lane detection, sign recognition, and collision avoidance will be installed in the vehicles.

By utilizing diverse AI networks dedicated to specific fields of expertise, the safety and stability of autonomous vehicles can be enhanced. NVIDIA’s development work has applied this advanced AI to the complex situation of driving. The BB8 autonomous vehicle demonstration can be seen in the following video.

More information about NVIDIA’s automation solutions, such as the AI autonomous driving supercomputer DRIVE PX2 and the open platform for developers NVIDIA DriveWorks, can be found at the following link (NVIDIA.com/drive). Research data on the method for identifying the priority process of deep neural networks can be found in the following white paper.