Autonomous driving object recognition, camera-lidar-radar useful
Deep Learning Object Recognition, Dataset Needed for AI Learning
Datasets can be supplemented with autonomous driving scenarios


True autonomous driving is achieved when the vehicle fully assumes the role of a driver.

Drivers perform a variety of actions while driving. Understanding the driving environment surrounding the vehicle is a particularly important role. And processing visual information is the process that accounts for the greatest proportion of understanding the driving environment.

Object recognition technology is the beginning of technologies that detect objects from visual information, plan a vehicle's path based on this, and drive the vehicle to move.

Professor Yong-gu Lee of the Department of Mechanical Engineering at the Gwangju Institute of Science and Technology (GIST) looked into the current status of autonomous vehicle object recognition technology and standards through his article, “Current Status of Object Recognition Technology for Autonomous Vehicles,” which was contributed to the 191st issue of the Telecommunications Technology Association (TTA) Journal.


Sensors required for autonomous vehicle object recognition
Professor Lee Yong-gu said, “Unlike object recognition in other fields, object recognition in the autonomous vehicle field utilizes diverse and heterogeneous sensors simultaneously,” and “The reason simultaneous use is necessary is because satisfactory object recognition cannot be achieved with information from only one sensor.”

It is difficult to deal with failures with a single sensor, and the recognition rate of a single sensor is not yet reliable enough, so it is necessary to combine information from various sensors to increase the recognition rate.

Collect visual informationThe most representative sensor is the camera. The camera can collect a wealth of information, but it requires multiple calculations to recognize objects. At this time, static digital photo recognition and dynamic image recognition must be performed simultaneously. In addition, since the camera collects information from light reflected from the subject by external lighting, the quality of the information is likely to deteriorate in bad weather.

Therefore, autonomous vehicles need additional sensors to detect their surroundings in addition to cameras. The professor mentioned Lidar and Radar as representative sensors. Both sensors transmit electromagnetic waves to the outside and collect signals scattered by objects.

Lidar is suitable for use in autonomous vehicles because it processes and provides three-dimensional information. Radar provides two-dimensional information, so the information dimension is low, but it can quickly and reliably identify collision risk situations, so it is useful for detecting critical and sudden situations. Lidar uses a laser light source in the far-infrared band of 1000 nm or more, and radar uses a radio frequency of 10 m to 200 m.

Radar is easy to process reflected signals, especially because it uses a single-wavelength light source. This is because the wavelength of the light source is smaller than the object that the autonomous vehicle must recognize, so the light source is regarded as a ray that travels straight and reflects, and can be interpreted as a very simple physical phenomenon. In addition, it is cheap and its electromagnetic waves are not absorbed by fog or rain, so it is adopted by many autonomous vehicles.

In addition, there is an ultrasonic sensor that is widely used as a sensor that emits a warning sound when approaching a nearby vehicle while parking a car. The ultrasonic sensor generates mechanical sound waves and measures the reflection from surrounding objects to calculate the distance. It operates without error even in bad weather, and it even works well in bad conditions where the sensor is covered with dust. However, it has not yet been widely adopted in autonomous vehicles.


Process of collecting visual information using cameras, lidar, and radar
The information that the camera obtains is the intensity of the three wavelengths of visible light of the subject projected perpendicularly to the camera. With the popularization of digital cameras, the information that is usually obtained is arranged in two directions perpendicular to the projection direction.

To recognize objects from multichannel array information, the region where the objects are located must be cut out (region selection) and the objects in that region must be classified. To speed up the calculation, many network architectures perform these two tasks simultaneously.

Autonomous driving requires object recognition not only in still images but also in continuous images. It also requires calculations to assign IDs to each object in the moving images and track them. This tracking is difficult due to occlusion between objects. When a vehicle equipped with a camera moves at high speed, the time to detect objects is also limited.

Using two cameras, we obtain the three-dimensional coordinates of the subject. Autonomous vehicles must not only recognize objects, but also determine whether objects are far or close, and pay more attention to nearby objects.

For efficient computation, objects that are outside a certain distance are usually not detected. “In autonomous driving technology that does not use lidar, such as Tesla and Mobileye, three-dimensional location information about objects on the road is obtained in this way,” the professor added.

In the case of autonomous vehicles using lidar, distance information of the subject can be obtained from the sensor, making it easy to distinguish between distance and distance. When using lidar, processed data that expresses the subject surface as a point cloud is used rather than using raw data. Here, the points become the coordinate values x, y, and z of the Cartesian coordinate system with the lidar sensor coordinate system as the origin.

When recognizing objects in point clouds, a network consisting of a multilayer perceptron with a transformation preprocessor specialized for point clouds called 'PointNet' is mostly used.

The preprocessor of PointNet uses the characteristics that the point cloud is order-independent, the points must be distinguished by distances, and they must be independent of transformations such as rotation or translation. Using PointNet and 'PointNet++', which gives it a hierarchical structure, it is possible to segment the surrounding background as well as recognize objects.

The disadvantage of lidar is that the point cloud it obtains is very small, so when it gets to a certain distance from the sensor, only a small number of point clouds are obtained, making object recognition difficult. To supplement this, the color image obtained from the camera is augmented around the point cloud of the lidar to supplement the data in the unmeasured area.

In such cases, other sensor information must be fused. Fusing data between multiple sensors is not only to account for the characteristics of the sensors, but also to prepare for cases where some sensors do not operate smoothly.


Requirements for autonomous driving datasets for deep learning training
Deep learning-based object recognition is a technology that trains AI models with large datasets to find objects from information obtained from sensors such as cameras, lidar, and radar.

Many autonomous driving datasets currently being released are multi-modal, collected simultaneously by multiple sensors attached to the vehicle. They often include LiDAR point clouds, and more recently, there are also cases that provide radar measurements.

Due to this multimodal nature, the number of scenes used as shooting units generally does not exceed 1,000, and the number of classes used to distinguish objects is also around 10. Common classes include cars, pedestrians, and bicycles.

The autonomous driving dataset must basically include data that fits the characteristics of each mode.

For camera data, RGB (Red, Green, Blue) images are required as a basic requirement. In addition, information such as lens, mounting location, shooting time, GNSS (Global Navigation Satellite System) information, and climate can be included to further improve network accuracy. The class and bounding box of the detected object must also be present.

However, no standard has been established to express this information. “There is no common agreement on autonomous vehicles in the industry yet, so no standards have been created,” the professor said. “When creating a dataset for deep learning, in addition to a standardized classification system like the above, separate norms for the rules for actually applying bounding boxes are also needed.”

The dataset for object recognition does not necessarily have only ideal photos. Object recognizers must work in real-world situations, and often the images are not completely understood. The professor advised, “Even if the dataset is not perfect, creating a bounding box for the object contributes to creating a detector with good performance.”

In some images, there may be exceptional situations such as occlusion, light reflection, and blur. For labeling such cases, it is suggested that indicating a comprehensive class as much as possible and indicating the reason for the difficulty in identification will lead to creating a good detector.


Helps build comprehensive autonomous driving scenarios and supplement datasets
Autonomous vehicles need to not only recognize objects, but also understand how objects move over time. Many recent datasets provide a large number of sensor measurements over scenes of the order of tens of seconds. The fact that these scenes encompass all the necessary traffic situations is linked to the safety of autonomous vehicles.

In response to the industry's demand for this, description languages that can describe scenes are being published. Currently, such research is being conducted in Korea, but it is described in natural language rather than in a computer-readable form.

There are also tools that use virtual reality techniques to create situations that are difficult to reproduce in real life as a way to express scenes. They use realistic graphic environments and physics-based methods that are also used in game technology to create datasets by calculating measurements from various sensors to supplement insufficient datasets.