Verification of real-time embedded object recognition and tracking, and localization system using a multirotor

Abstract

As the usage of unmanned aircraft increases, a vision-based mission becomes one of the important topics in the unmanned aircraft research. SAR and EO/IR sensors can be readily equipped with the unmanned aircraft for the vision-based missions. Especially, the EO/IR sensors are widely used because they are easy to apply to small unmanned aircraft and are inexpensive. There are many vision-based studies such as object recognition, object tracking, vision-based navigation, and object localization. Object recognition, especially, is known as one of the most difficult research in the computer vision and unmanned aircraft field because it has many different cases depending on the flight environments. Therefore, there have been various studies on the object recognition such as SIFT and SURF. Because the SIFT and SURF have excessive memory usage and computational load, however, it is tough to apply them to a real-time system. To overcome the limitations, binary descriptor based studies have been such as BRIEF, ORB, BRISK, and FREAK. These algorithms are easy to apply to a low-power device such as cell-phone or Digital Signal Processor (DSP) because they have a low memory usage and a low computational load. In this paper, the ORB is used for the object recognition and tracking. And it is implemented in an image processing computer that has TMS320DM648 as the main processor. To increase the image processing performance, we use the assembly language based vision library. Because the ORB is rotation-invariant but is not scale-invariant, the object template is implemented to various scales. Also, to reduce the memory usage of the image processing computer, the all binary descriptors of the templates are saved as a database. A flight control computer that has TMS320F28346 as the main processor calculates the object location from the sensory data of GPS, INS, gimbal, and pixel position of the recognized object. Also, a linear Kalman filter is adopted to estimate the object position. To verify the object recognition and tracking system, and localization system, a flight experiment is conducted by using an unmanned aircraft that has a gimbal and an EO-camera. The flight experiment result shows that the implemented object recognition and tracking algorithm based on ORB works well and can operate in real-time. Also, the result shows that the Kalman filter estimates the object position well.