臺灣博碩士論文加值系統

本論文旨在改善機器人作業系統「導航」功能裡區域路徑規劃演算法面對高度動態變化之環境時無法有效避障的問題。在機器人作業系統中，區域路徑規劃演算法官方係使用Dynamic Window Approach (DWA)，該方法實作簡易並且通常能有效繞過靜態障礙物。然在實際進行多次測試後發現，該方法在高度動態變化之環境的避障效果不佳。此問題主因為DWA僅針對當前所探測之障礙物進行迴避路線規劃，並未預測障礙物接下來可能的動態行為。
典型應用裡，機器人作業系統使用光學雷達來感測機器人與周遭物體的相對距離，並提供當下的感測資料給DWA運算。本研究之創意在於利用近期內的距離感測資料和神經網路預測障礙物的可能移動方式，以提升避障的成功率。由於依照時間順序所感測之距離資料為一種序列式資料，故本研究採用Gated Recurrent Unit (GRU)來進行訓練和預測，並搭配Twin Delayed Deep Deterministic Policy Gradient (TD3)深度增強式學習演算法來訓練神經網路，訓練過程分為兩階段：第一階段採用「模仿學習」搭配「深度增強式學習」的方式讓神經網路可學習到DWA演算法之優點；第二階段則是單獨以「深度增強式學習」訓練神經網路來學習遇到動態障礙物時所能採取的最佳閃避動作。模擬實驗結果顯示，使用本方法進行避障規劃後，機器人之行動流暢度雖略遜於DWA，但能有效預測動態障礙物之行動並提早進行避障，某些無法透過單純改變方向閃避障礙的情況下還可主動後退以提供足夠之空間來閃避障礙物。

This thesis focuses on solving the problem that the local path planning algorithm of Robot Operating System (ROS) can’t cope with a highly dynamic environment. In ROS, the official local path planning algorithm is called Dynamic Window Approach (DWA), which is simple to implement and effective to bypass static obstacles in many cases. However, it is observed that DWA is ineffective to bypass moving obstacles. The key reason is that DWA takes into account only the current positions of the obstacles and thus, does not predict their movements in the coming periods.
In typical use cases, ROS adopts a LiDAR to measure the distances from an Autonomous Mobile Robot (AMR) to the surrounding environment and the latest measurements are provided to DWA. Compare to the existing approach, the idea behind this thesis is to adopt a Neural Network with the recent distance measurements to predict the movements of the moving obstacles for increasing the success ratio of the obstacle avoidance. Since the distance measurements are a type of time series data, Gated Recurrent Unit (GRU)and Twin Delayed Deep Deterministic Policy Gradient (TD3) are adopted to train the model for local path planning.
The training process is composed of two phases: (1) Imitation learning as well as deep reinforcement learning and (2) reinforcement learning. The phase 1 is to learn the advantages of DWA while the phase 2 is to learn the best decision to bypass the dynamic obstacles.
The simulation results show that the proposed method can effectively predict the following movements of the moving obstacles for executing the obstacle avoidance process in advance at the cost of a slightly less smooth movement than DWA. Furthermore, in some cases that there is no room for both a traditional AMR and a moving obstacle to pass each other, the proposed method demands the AMR to make way for the moving obstacle thus solving the problem.

摘要...i
Abstract...ii
誌謝...iii
目錄...iv
表目錄...vi
圖目錄...vii
第一章 緒論...1
1.1 研究背景...1
1.2 研究動機...1
1.3 研究簡述...1
1.4 論文架構...2
第二章 文獻探討與相關技術...3
2.1 文獻探討...3
2.2 機器人作業系統(Robot Operating System)...3
2.3 Dynamic Window Approach (DWA)...4
2.4 深度增強式學習(Deep Reinforcement Learning, DRL)...5
2.4.1 深度增強式學習簡介...5
2.4.2 深度增強式學習之種類...7
2.5 Actor-Critic架構...11
2.6 Twin Delayed Deep Deterministic Policy Gradient (TD3)...13
2.6.1 Q值高估問題(overestimate)...13
2.6.2 TD3演算法之更新方式...15
2.6.3 軟更新(Soft update)...16
2.7 部分可觀察馬可夫決策過程...16
2.8 循環神經網路(Recurrent Neural Network)...17
2.9 模仿學習(Imitation Learning)...19
2.9.1 反向增強式學習(Inverse Reinforcement Learning, IRL)...19
2.9.2 行為模仿(Behavior Cloning)...20
第三章 研究內容與方法...21
3.1 模擬環境與獎勵函數...21
3.1.1 高度動態變化之模擬環境...21
3.1.2 獎勵函數(Reward function)...23
3.2 TD3與RNN結合之模型架構...27
3.3 應用模仿學習...30
第四章 實驗與評估...32
4.1 實驗套件介紹...32
4.2 實驗環節與應用參數...32
4.2.1 單獨應用TD3之高度動態變化環境實驗...33
4.2.2 應用TD3與GRU之高度動態變化環境實驗(無模仿學習)...34
4.2.3 應用TD3與GRU之高度動態變化環境實驗(具模仿學習)...35
4.3 避障評估...36
第五章 結論與未來展望...37
參考文獻...38
附錄一...40
Extended Abstract...41

[1]機器人作業系統, available from: https://www.ros.org.
[2]Base local planner, available from: http://wiki.ros.org/base_local_planner.
[3]Dieter Foxy, Wolfram Burgardy, Sebastian Thrun, 1997, “The Dynamic Window Approach to Collision Avoidance,” IEEE Robotics & Automation Magazine, vol. 4, issue 1, pp. 23-33, March.
[4]DWA local planner, available from: http://wiki.ros.org/dwa_local_planner.
[5]Scott Fujimoto, Herke van Hoof, David Meger, 2018, “Addressing Function Approximation Error in Actor-Critic Methods”, International Conference on Machine Learning, 1582-1591, February.
[6]Talebpour, Zeynab, Martinoli, Alcherio, 2018, “Risk-Based Human-Aware Multi-Robot Coordination in Dynamic Environments Shared with Humans”, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3365-3372.
[7]Qingyang Tan, Tingxiang Fan, Jia Pan, Dinesh Manocha, 2020, “DeepMNavigate: Deep Reinforced Multi-Robot Navigation Unifying Local & Global Collision Avoidance”, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA (Virtual), October 25-29.
[8]Xiaoyun Lei, Zhian Zhang, Peifang Dong, 2018, “Dynamic Path Planning of Unknown Environment Based on Deep Reinforcement Learning”, Journal of Robotics, vol. 2018, September.
[9]Matej Dobrevski, Danijel Skočaj, 2020, “Adaptive Dynamic Window Approach for Local Navigation”, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6930-6936, February.
[10]Dainel Zhang, Colleen P. Bailey, 2020, “Obstacle Avoidance and Navigation Utilizing Reinforcement Learning with Reward Shaping”, arXiv: 2003.12863, March.
[11]Herrn Anton Maximilian Schafer, 2008, “Reinforcement Learning with Recurrent Neural Networks”, University of Osnabruck, Institute for Computer Science Neuroinformatics Group.
[12]Lu Wang, Wei Zhang, Xiaofeng He, Hongyuan Zha, 2018, “Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation”, Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp.2447-2456, July.
[13]Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, Will Dabney, 2019, “Recurrent Experience Replay in Distributed Reinforcement Learning”, International Conference on Learning Representations, January.
[14]Dynamic Programming, available from: https://en.wikipedia.org/wiki/Dynamic_programming.
[15]Supervised Learning, available from: https://en.wikipedia.org/wiki/Supervised_learning.
[16]Deep Mind AlphaGo, available from: https://deepmind.com/research/case-studies/alphago-the-story-so-far.
[17]Microsoft Bonsai, available from: https://docs.microsoft.com/en-us/bonsai/product.
[18]What is the difference between model-based and model-free reinforcement learning? ,available from: https://www.quora.com/What-is-the-difference-between-model-based-and-model-free-reinforcement-learning.
[19]RL survey, available from: https://github.com/AI4Finance-LLC/ElegantRL/blob/master/figs/RL_survey_2020.pdf.
[20]Mean Square Error, Available from: https://en.wikipedia.org/wiki/Mean_squared_error.
[21]Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra, 2016, “Continuous control with deep reinforcement learning”, arXiv: 1509.02971, September.
[22]Van Hasselt, Hado, 2010, “Double Q-learning”, pp. 2613-2621, January.
[23]Markov Decision Process, available from: https://en.wikipedia.org/wiki/Markov_decision_process.
[24]Matthew F. Dixon, Igor Halperin, Paul Bilokon, 2020, “Inverse Reinforcement Learning and Imitation Learning”, Machine Learning in Finance, Springer, Cham.
[25]Felipe Codevilla, Eder Santana, Antonio Lopez, Adrien Gaidon, 2019, “Exploring the Limitations of Behavior Cloning for Autonomous Driving”, 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9328-9337.
[26]ROBOTIS-Turtlebot3. Available from: https://emanual.robotis.com/docs/en/platform/turtlebot3/overview.
[27]turtlebot3_simulations open source code. Available from: https://github.com/ROBOTIS-GIT/turtlebot3_simulations.
[28]Shreyansh Daftry, J. Andrew Bagnell, Martial Hebert, 2016, “Learning Transferable Policies for Monocular Reactive MAV Control”, arXiv: 1608.00627, August.
[29]車聯網, available from: https://zh.wikipedia.org/wiki/%E8%BB%8A%E8%81%AF%E7%B6%B2.