臺灣博碩士論文加值系統

近年來，無人駕駛飛機數量的增加已使交通管理望而卻步，尤其是在城市環境中。無人機群的安全運行和優化導航已成為越來越受關注的問題。在本文中，我們使用了深度強化學習並找到了一種空氣動力學的加速/減速無人機控制方法。一般來說，無人機在遭遇時的瞬時加速/減速允許他們的同伴首先或最後通過，但大多數都沒有考慮無人機本身的運動學。無人機本身的運動學在瞬時加速/減速引起的高度變化為避免碰撞提供了額外的路徑，以充分利用無人機群並最大限度地發揮其性能通過空中交通路口。首先，無人機群之間的相互通信是通過無人機互聯網建立的，以發送和接收無人機群的當前狀態並檢測碰撞，以利用深度強化學習尋找到不可避免的高度變化，在障礙物的情況下通過空中交通路口。但無人機瞬時加速/減速的能量消耗是不容忽視的，所以無人機速度控制的正確設計對於無人機的電源管理至關重要。依照實際無人機紀錄的參考能量，採用深度強化學習進行加速/減速與能耗控制，避免任何風險的衝突以控制能量消耗達到最小化。此外，我們通過仿真分析參數，在控制任務中實現我們的方法，並證明該方法具有出色的旅行時間和功率管理性能。

In recent years, the increase in the number of drones has made traffic management prohibitive, especially in urban environments. The safe operation and optimized navigation of drone swarms have become more and more concerning issues. In this article, we used Deep Reinforcement Learning (DRL) and found an aerodynamic acceleration/deceleration drone control method. Generally speaking, the instantaneous acceleration/deceleration of drones during encounters allows their companions to pass first or last, but most do not consider the kinematics of the drone itself. The kinematics of the drone itself provides an additional path to avoid collisions due to the instantaneous acceleration/deceleration of the kinematics of the drone itself, to make full use of the drone group and maximize its performance through the air traffic intersection. First of all, the mutual communication between the drone swarms is established through the Internet of Drones(IoD) to send and receive the current state of the drone swarms and detect collisions, to use DRL to find the inevitable height changes, in the case of obstacles. Go down through the air traffic intersection. However, the energy consumption of the drones instantaneous acceleration/deceleration cannot be ignored, so the correct design of the drones speed control is crucial to the drones power management. According to the reference energy recorded by the actual drone, DRL is used for acceleration/deceleration and energy consumption control, avoiding any risk of conflict and minimizing energy consumption. In addition, we use simulation to analyze the parameters to implement our method in the control task and prove that the method has excellent travel time and power management performance.

Chinese Abstract.............................................................i
English Abstract.............................................................iii
Acknowledgements.............................................................v
Contents.....................................................................vi
List of Tables...............................................................viii
List of Figures..............................................................ix
1 Introduction...............................................................1
2 Literature Review..........................................................6
2.1 Non-­reinforcement Learning or Even Other Methods of Non-­­Learning.........6
2.2 Deep Learning Collision Avoidance Algorithms.............................7
3 Motivation and Problem Definition..........................................10
3.1 Drone and Obstacle Drone Collision Chance Constraints....................14
3.2 Communication Commands Delay.............................................14
3.3 Drone Energy Consumption.................................................16
3.4 Model Predictive Control.................................................17
4 Algorithms.................................................................18
4.1 Deep Q Learning..........................................................18
4.2 Drone Reference Altitude Position Curve and Reference Energy Table.......21
4.3 Reward Design ...........................................................26
5 Performance Evaluation.....................................................28
5.1 Experimental Results.....................................................31
5.1.1 Different reward tendencies...........................................31
5.1.2 The distance between the drone and the intersection is different......32
5.1.3 Different command distance............................................33
5.1.4 Comprehensive comparison..............................................34
6 Conclusion.................................................................37
References...................................................................40

[1] Menouar, Hamid, et al. "UAV-enabled intelligent transportation systems for the smart city: Applications and challenges." IEEE Communications Magazine 55.3 (2017): 22-28.
[2] Alladi, Tejasvi, et al. "Applications of blockchain in unmanned aerial vehicles: A review." Vehicular Communications 23 (2020): 100249.
[3] Wang, Desheng, et al. "Routing and scheduling for hybrid truck-drone collaborative parcel delivery with independent and truck-carried drones." IEEE Internet of Things Journal 6.6 (2019): 10483-10495.
[4] Gharibi, Mirmojtaba, Raouf Boutaba, and Steven L. Waslander. "Internet of drones." IEEE Access 4 (2016): 1148-1162.
[5] Resat, H. Giray. "Design and analysis of novel hybrid multi-objective optimization approach for data-driven sustainable delivery systems." IEEE Access 8 (2020): 90280-90293.
[6] Verri, Filipe Alves Neto, et al. "An analysis on tradable permit models for last-mile delivery drones." IEEE Access 8 (2020): 186279-186290.
[7] Kopardekar, Parimal, et al. "Unmanned aircraft system traffic management (UTM) concept of operations." 16th AIAA Aviation Technology, Integration, and Operations Conference. AIAA, 2016.
[8] Arul, Senthil Hariharan, and Dinesh Manocha. "Dcad: Decentralized collision avoidance with dynamics constraints for agile quadrotor swarms." IEEE Robotics and Automation Letters 5.2 (2020): 1191-1198.
[9] Arul, Senthil Hariharan, et al. "LSwarm: Efficient collision avoidance for large swarms with coverage constraints in complex urban scenes." IEEE Robotics and Automation Letters 4.4 (2019): 3940-3947.
[10] Bahabry, Ahmed, et al. "Low-altitude navigation for multi-rotor drones in urban areas." IEEE Access 7 (2019): 87716-87731.
[11] Steinberg, Andrew, Mihaela Cardei, and Ionut Cardei. "UAS Batch Path Planning With a Space-Time Graph." IEEE Open Journal of Intelligent Transportation Systems 2 (2021): 60-72.
[12] Minucci, Franco, Evgenii Vinogradov, and Sofie Pollin. "Avoiding collisions at any (low) cost: ADS-B like position broadcast for UAVs." Ieee Access 8 (2020): 121843-121857.
[13] Morozs, Nils, Paul D. Mitchell, and Roee Diamant. "Scalable adaptive networking for the Internet of underwater things." IEEE Internet of Things Journal 7.10 (2020): 10023-10037.
[14] Zou, Yiyuan, et al. "Fast Collision Detection for Small Unmanned Aircraft Systems in Urban Airspace." IEEE Access 9 (2021): 16630-16641.
[15] Arul, Senthil Hariharan, and Dinesh Manocha. "Swarmcco: Probabilistic reactive collision avoidance for quadrotor swarms under uncertainty." IEEE Robotics and Automation Letters 6.2 (2021): 2437-2444.
[16] Yao, Jingjing, and Nirwan Ansari. "Online task allocation and flying control in fog-aided internet of drones." IEEE Transactions on Vehicular Technology 69.5 (2020): 5562-5569.
[17] Park, Jungwon, and H. Jin Kim. "Online Trajectory Planning for Multiple Quadrotors in Dynamic Environments Using Relative Safe Flight Corridor." IEEE Robotics and Automation Letters 6.2 (2020): 659-666.
[18] Everett, Michael, Yu Fan Chen, and Jonathan P. How. "Collision avoidance in pedestrian-rich environments with deep reinforcement learning." IEEE Access 9 (2021): 10357-10377.
[19] Zhu, Jiangcheng, et al. "Hierarchical decision and control for continuous multitarget problem: Policy evaluation with action delay." IEEE transactions on neural networks and learning systems 30.2 (2018): 464-473.
[20] Qie, Han, et al. "Joint optimization of multi-UAV target assignment and path planning based on multi-agent reinforcement learning." IEEE access 7 (2019): 146264-146272.
[21] Kim, Myounghoe, et al. "Unexpected collision avoidance driving strategy using deep reinforcement learning." IEEE Access 8 (2020): 17243-17252.
[22] Meyer, Eivind, et al. "COLREG-compliant collision avoidance for unmanned surface vehicle using deep reinforcement learning." IEEE Access 8 (2020): 165344-165364.
[23] Wang, Dawei, et al. "A two-stage reinforcement learning approach for multi-UAV collision avoidance under imperfect sensing." IEEE Robotics and Automation Letters 5.2 (2020): 3098-3105.
[24]Yang, Zhaoyang, et al. "Hierarchical deep reinforcement learning for continuous action control." IEEE transactions on neural networks and learning systems 29.11 (2018): 5174-5184.
[25] Pfeiffer, Mark, et al. "Reinforced imitation: Sample efficient deep reinforcement learning for mapless navigation by leveraging prior demonstrations." IEEE Robotics and Automation Letters 3.4 (2018): 4423-4430.