Sky Command

■ Project Introduction

(Problem Definition)

The problem with modern tactical unmanned robots is that despite advanced technology, control methods still rely on manual operation. To deploy existing military unmanned vehicles in operations, significant time and cost are spent training control operators. Additionally, in real combat, having the operator carry a controller and pilot while viewing a first-person perspective is not only difficult, but also results in loss of combat effectiveness such as neglect of forward observation.

(Intuitive Interface Found in StarCraft)

We have experienced and trained with a very convenient and efficient unmanned resource control method through screens for decades. Real-time strategy games like StarCraft move tactical units with a single mouse click on the minimap without complex joysticks. Project Cheonan: Sky Commander was planned to establish a 'single-click based tactical control interface' by applying popular and intuitive user experience to real military technology.

(Realization of Tactical Minimap)

Aerial vision sensing technology and ground actuator control and algorithms are organically combined to provide operators with convenience and an intuitive HRI environment. When a reconnaissance observer (web camera) in the airspace above the operation area captures the battlefield, the overhead third-person perspective video is displayed on the operator's (central manager's) monitor. This video is projected in the form of a 'minimap' within a game, allowing a wide field of view to be secured in the operation area. When the operator (central manager) clicks on a region requiring support or an allied position on the monitor screen, OpenCV converts 2D pixel coordinates on the screen to coordinates in real space in real-time and transmits them to the robot. Upon receiving these coordinates, the robot uses the A* algorithm and object detection through AI learning models to autonomously bypass obstacles and move to the target point. Therefore, the operator (central manager) is freed from the burden of manual machine operation, allowing them to focus on the military's tactical decision-making and operations instead of investing time and cost in equipment control, creating an optimal environment.

(A* Algorithm and PID Control)

The overhead central camera (web camera) positioned at the top is utilized like a satellite or drone to secure the entire image of the operation area and implement destination setting by mouse click. When the destination is entered, the A* algorithm calculates in real-time the shortest path that avoids obstacles (blue cones) such as enemy barricades or minefields. Subsequently, the current position of the robot is identified using ArUco markers, and PID control is applied to drive precisely to the target point without deviating from the path.

(Optimization of Mechanical Avoidance Maneuvers)

Initially, it was designed so that when an ultrasonic sensor in the front detects a sudden obstacle, the A* algorithm would be rerun to recalculate the path. However, severe frame drop phenomena were observed during the recalculation process. To resolve this, instead of software-based path recalculation, a mechanical physical avoidance logic was created where the robot reverses for a certain period and then rotates by a specific angle. This significantly reduced system computational load, increasing avoidance and safety in sudden situations.

(AI Model Learning for Blind Spot Concealment Threat Identification)

Using only the overhead central camera (web camera) and ultrasonic sensor makes it difficult to identify traps such as anti-personnel mines hidden outside the camera's field of view or enemy ambush situations. To identify situations existing outside the camera's view, a camera module was attached to a Raspberry Pi to inspect areas not visible to the overhead central camera (web camera—satellite or drone), such as caves and interiors. Additionally, the existing OpenCV HSV color recognition had limitations in distinguishing allied and enemy forces based solely on color. To overcome this, the Roboflow platform was used to build and train a YOLOv8 dataset with cremoroa (threat elements) labeled as 'Enemy' and soldier figures (protection targets) labeled as 'Ally', acquiring object detection capability. Furthermore, when cremoroa is detected, an alarm sound is transmitted to the control monitor along with a real-time UI animation that operates on the map, completing a complete system.

(ONNX Quantization)

To run the trained deep learning model without latency in the limited Raspberry Pi 5 onboard environment, model weights were converted to ONNX format for quantization. As a result, a system was completed that enables real-time monitoring without latency in response to sudden situations.

(Multi-Actuator Based Tactical Execution)

Beyond simply achieving situational awareness, an embedded control system was built to translate judgment results into physical actions on actual hardware. When the AI model identifies isolated allied soldiers, GPIO signals from the Raspberry Pi activate relay modules. When needed, a humidifier module deploys smoke screens to block enemy vision, and the electromagnet module power is shut off to drop supplies, enabling complex tactical scenarios to be executed without delay as designed.

(Real-Time Remote Control UI and Alarm System)

A control system was configured to enable remote operators to understand the object detection results from deployed robots in the field. When onboard AI detects threats (cremoroa) or allies (soldiers), they are transmitted to the control monitor via wireless communication, and the control infrastructure was completed with UI animation and alarm sound output.

(Establishment of Dual Camera Hybrid Control System)

To overcome the limitation of being able to drive only in areas visible to the overhead central camera (web camera), a dual vision system was adopted. Unlike the overhead central camera (web camera) dedicated to AI computation, a Raspberry Pi dedicated camera was configured independently to provide operators with unrestricted, seamless real-time streaming vision. When emergencies occur that cannot be controlled by the overhead central camera (web camera), to resolve them, operators can immediately take control with the WASD keyboard without latency and operate manually, with this functionality added.

(Frame Mounting and Hardware Integration Optimization)

To mount multiple cameras, humidifiers, breadboards, Raspberry Pi, and other components in a limited space, a 3D printer was used to design and manufacture custom frames, integrating the hardware configuration.

(Poor ArUco Marker Recognition Rate)

Initially, the plan was to guide the RC car based on drone GPS data, but due to budget constraints and limitations of indoor test spaces, the system architecture was redesigned. Accordingly, the system was converted to a vision-based control system where the RC car position is tracked in real-time using ArUco markers with the top web camera, and movement commands are issued from the central control PC. However, technical limitations arose due to characteristics of general wide-angle web cameras: insufficient light in low-light environments, motion blur occurring during driving, and dramatic decrease in long-distance recognition rate. To overcome this, a 4-stage image preprocessing pipeline was established that adjusts brightness, gamma, contrast, and saturation step-by-step. As a result, continuity and robustness of marker tracking could be secured even in changing lighting conditions and long-distance driving environments.

(Frame Drop Due to Ultrasonic Sensor Computational Overload)

Initially, when an ultrasonic sensor detected a sudden obstacle during driving, the robot's response speed decreased dramatically and video frames became choppy. When a sudden obstacle was detected, the system was designed to conduct path research through the A* algorithm. The cause was that real-time A* recalculation in the Raspberry Pi environment was causing overload. Therefore, software-based path recalculation was changed to hardware-based physical avoidance. When obstacles enter within a certain distance of the ultrasonic sensor, motor control is applied to avoid collision: reverse for a certain period and then rotate by a specific angle, with this maneuver logic applied in code. This dramatically reduced system computation, preventing serious frame drop and enabling immediate obstacle (blue cone) avoidance.

(Limitations of HSV Object Recognition)

Initially, OpenCV HSV color masking technique was used for sudden situations and enemy detection. However, the method was vulnerable to lighting changes and frequently produced errors in identifying objects with complex shapes. To resolve this, deep learning object recognition was applied. Image data was collected with cremoroa labeled as 'Enemy' and soldier figures labeled as 'Ally' using Roboflow, the YOLOv8 model was trained, and for smooth processing on Raspberry Pi, it was quantized to ONNX format and installed. Therefore, an AI system was built that accurately detects without depending on changes in lighting or angles through real-time edge inference.

■ Major Achievements and Characteristics

1. Autonomous Driving Path Generation Based on PID Control and A* Algorithm

• Real-time position and angle tracking based on ArUco markers and PID control were implemented to reduce errors between the linear section path and actual path. Additionally, by applying the A* algorithm, the robot draws a path avoiding obstacles that are recognized along the straight line to the target point, and operates in the sequence of aligning angles according to the drawn path and moving straight following the path search line. In other words, PID control converges the difference between the path and ArUco marker recognition angle to zero, and once the angle is aligned, the distance (error) between the ArUco marker and target point is converged to zero as the robot moves toward the target.

When the A* algorithm is applied to PID control, the optimal path avoiding obstacles to the target point is searched, and the robot moves while converging angle difference and distance to zero following the line as shown in the photo below.

2. Dual Vision-Based Hybrid (Autonomous + Manual) Control System

Since the overhead central camera (web camera) recognizes ArUco markers, uncontrollable situations can arise when ArUco markers are not recognized due to camera angle issues, fog, or blind spots. To prepare for this situation, a manual operation mode was added that converts the front camera (Pi camera module) to a first-person perspective for direct manipulation. Since conversion is done through the UI in the lower left portion of the central control monitor, rapid switching and response are possible even in urgent situations.

3. Real-Time Object Recognition (YOLOv8) and Tactical Proactive Response

A custom dataset based on Roboflow (Enemy (cremoroa) and Ally (soldiers)) was trained with the YOLOv8 model and quantized to ONNX format to complete a real-time object detection environment.

Simultaneously with AI identifying Enemy or Ally, the location of the corresponding object is visualized in real-time on the central control monitor map, and alarm sounds and UI animation notifications are implemented to intuitively deliver information.

■ Future Improvement Plans

<Multi-Unit Simultaneous Control>

During the planning stage, a plan was established to control at least 2-3 RC cars, but only one RC car could be produced due to budget constraints. Currently, a control interface for a single robot has been built, but in the future, like 'StarCraft', multiple robots will be controlled by dragging with the mouse to assign coordinates at once, and the system will be expanded into a multi-agent swarm driving control network where robots share positions with each other to maintain formation and move.

<Mapping Technology for Standalone Robots>

The current project uses a global vision method where an overhead control camera positioned in the air overlooks the entire terrain and directs the path. While this may be effective in wide field environments, it has limitations where autonomous driving is impossible in unknown environments such as indoor urban areas, bunkers, etc., where drone or satellite vision cannot reach. To overcome this limitation, control through the Raspberry Pi camera that was previously used is possible outside the control camera's area, but has the limitation of not knowing the surrounding situation and only seeing what is in front. However, if a small 2D LiDAR sensor camera is mounted on the robot body itself, this limitation can be overcome. By introducing SLAM technology that scans and maps the surroundings independently to identify location, the system will be upgraded to a fully autonomous reconnaissance platform capable of self-mapping and exploration beyond manual operation using a Raspberry Pi dedicated camera in areas unreachable by the control camera.

<Implementation of Air-Ground Complex Control System>

The initial plan for this project was to actually deploy a drone to secure a global field of view. At the current stage, a fixed overhead web camera is implemented as an aerial command post role. In the future, actual small reconnaissance drones and ground reconnaissance robots will be linked through a wireless communication network. This will evolve so that the drone flies over the operation area in real-time, scans the map and transmits targets, and the ground robot receives them and executes tactics.

https://github.com/HOMEYONGJUNE/Sky-Command (Source code is submitted on GitHub due to file size limitations)