UAV Maneuvering Decision-Making Algorithm Based on Twin Delayed Deep Deterministic Policy Gradient Algorithm

Shuangxia Bai; Shaomei Song; Shiyang Liang; Jianmei Wang; Bo Li; Evgeny Neretin

doi:10.37965/jait.2021.12003

UAV Maneuvering Decision-Making Algorithm Based on Twin Delayed Deep Deterministic Policy Gradient Algorithm

Authors

Shuangxia Bai School of Electronics and Information, Northwestern Polytechnical University, China https://orcid.org/0000-0002-3710-9152
Shaomei Song Beijing Electro-Mechanical Engineering Insititute, China
Shiyang Liang Avic Luoyang Electro-optical Equipment Research Institute, China https://orcid.org/0000-0002-6941-2404
Jianmei Wang School of Electronics and Information, Northwestern Polytechnical University, China
Bo Li School of Electronics and Information, Northwestern Polytechnical University, China https://orcid.org/0000-0002-1415-4444
Evgeny Neretin School of Robotic and Intelligent Systems, Moscow Aviation Institute, Russian Federation https://orcid.org/0000-0003-0174-8929

DOI:

https://doi.org/10.37965/jait.2021.12003

Keywords:

air combat, DDPG, maneuvering decision-making, TD3

Abstract

Aiming at intelligent decision-making of unmanned aerial vehicle (UAV) based on situation information in air combat, a novel maneuvering decision method based on deep reinforcement learning is proposed in this paper. The autonomous maneuvering model of UAV is established by Markov Decision Process. The Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and the Deep Deterministic Policy Gradient (DDPG) algorithm in deep reinforcement learning are used to train the model, and the experimental results of the two algorithms are analyzed and compared. The simulation experiment results show that compared with the DDPG algorithm, the TD3 algorithm has stronger decision-making performance and faster convergence speed and is more suitable for solving combat problems. The algorithm proposed in this paper enables UAVs to autonomously make maneuvering decisions based on situation information such as position, speed, and relative azimuth, adjust their actions to approach, and successfully strike the enemy, providing a new method for UAVs to make intelligent maneuvering decisions during air combat.