臺灣博碩士論文加值系統

放觀世界對自動化的需求，可以注意到機器人開始被大量部署在貨物搬運、場內導覽與領位員等應用，而各場域往往礙於成本考量，其採用的電梯之尺寸只能供單台機器人搭乘。由於目前常見的電梯排程演算法幾乎都假設可讓多位乘客同時搭乘，且它們大多只考量乘客等待電梯時間(從呼叫電梯至進入電梯之經過時間) 以及搭乘時間(從搭乘電梯至抵達目標樓層之經過時間)，卻忽略「需優先搭乘的使用者」導致無法配合機器人之任務優先順序，因此這類現有的電梯排程演算法較不適合直接應用在各場域中機器人搭乘電梯之情境。為改善上述問題，本論文提出一種基於深度增強式學習的電梯排程演算法，可提升場域內複數機器人於電梯單乘載之跨樓層搬運時的搭乘效率，其降低機器人平均等待完成時間(從呼叫電梯至抵達目標樓層之平均經過時間)的同時也使接收緊急任務的機器人可以優先搭乘電梯。
本研究所提出的電梯排程演算法是採用Dueling Double Deep Q Network (D3QN) 演算法同時針對三種特性進行訓練，分別為機器人等待完成時間、電梯空載移動樓層數量和機器人所執行任務的優先權。訓練過程模擬真實各場域應用中的電梯所遭遇請求之情況，以使D3QN訓練結果在電梯的各種使用率(如：尖峰、離峰時刻)下皆可有效改善機器人搭乘電梯之效率。為驗證本演算法之效能，本論文採用Robotics Middleware Framework (RMF) 模擬電梯實際運行環境，實驗結果的效能量尺採用機器人平均等待完成時間、電梯平均空載移動樓層數量和排程結果平均違反優先權搭乘次數。實驗結果表明本研究使用D3QN演算法於各場域中機器人單乘載搭乘電梯之應用並針對上述三種特性進行排程，比相關研究使用LOOK演算法進行載運機器人比較後，雖然「空載移動樓層數」的表現略遜於LOOK演算法，但在「等待完成時間」與「違反優先權搭乘次數」的兩項效能量尺之結果得到改善。

In recent years, many mobile robots are deployed in warehouses or similar facilities and the demand for autonomous cross-floor navigation has accelerated considerably. Due to the high cost, elevators within such fields can typically be filled up with only one robot. However, contemporary elevator schedulers assume that an elevator car can carry several persons. Besides, the typical performance criteria optimized by those existing schedulers are the waiting time and the ride time. The waiting time is the time interval from the moment a hall call button is pressed by a passenger to the moment the passenger boards one car. The ride time is from when the passenger enters the car, until he arrives at his floor, including any intermediate stop. Since the priorities of the passengers are not considered for car allocation in those scheduling algorithms, they work poorly with robots performing tasks with multiple levels of priority. To this end, this thesis proposes an elevator scheduling algorithm based on Deep Reinforcement Learning for mobile robot transportation between floors with an elevator that can carry a maximum of only one robot. The proposed scheduler not only decreases the journey time including waiting time and the ride time but also enables the robot with higher priority to take the elevator earlier than the one with lower priority.
This thesis adopts the Dueling Double Deep Q Network, D3QN, to train the elevator scheduler for car allocation. The rewards for the car allocation decision are estimated based on the journey time, the number of floors an empty car traverses, and how the car allocation meets the robots’ priorities. The training environment simulates several types of incoming traffic, such as up peak and down peak, to improve the efficiency of the mobile robot transportation between floors. For performance evaluation, Robotics Middleware Framework (RMF) is adopted to emulate the actual operating environment with an elevator and multiple robots. The experimental results show that the proposed algorithm reduces the journey time and better meets the robots’ priorities at the cost of a higher number of floors an empty car traverses.

摘要...i
Abstract...ii
誌謝...iii
目錄...iv
表目錄...vi
圖目錄...vii
第一章 緒論...1
1.1 研究背景...1
1.2 研究動機...1
1.3 研究簡述...2
1.4 論文架構...2
第二章 相關研究與技術...3
2.1 相關研究...3
2.2 第二代機器人作業系統 (Robot Operating System 2)...4
2.2.1 第二代機器人作業系統之開發理念...4
2.2.2 第二代機器人作業系統之通訊特性...6
2.2.3 第二代機器人作業系統之特色...7
2.3 Robotics Middleware Framework (RMF)...9
2.4 深度增強式學習 (Deep Reinforcement Learning, DRL)...11
2.4.1 深度增強式學習簡介...11
2.4.2 深度增強式學習之應用類型...12
2.4.3 深度增強式學習之訓練方式...15
2.4.4 深度增強式學習之優化方式...17
2.5 Dueling Double Deep Q Network (D3QN)...18
2.5.1 Deep Q Network (DQN)...18
2.5.2 Double Deep Q Network (DDQN)...19
2.5.3 Dueling Double Deep Q Network (D3QN)...20
第三章 研究內容與方法...22
3.1 本研究系統架構...22
3.2 模擬環境...23
3.2.1 簡易模擬環境...23
3.2.2 RMF模擬環境...23
3.2.3 本論文之運作架構流程介紹...26
3.3 獎勵函數(Reward function)...29
3.4 本研究之神經網路架構...32
第四章 實驗與評估...35
4.1 實驗套件介紹...35
4.2 實驗環節與應用參數...35
4.2.1 本論文模型之訓練與測試結果...37
4.2.2 相關研究LOOK演算法...40
4.3 相關研究之效能比較...42
第五章 結論與未來展望...44
參考文獻...45
附錄一...47
附錄二...48
Extended Abstract...49

[1]“中光電智能機器人 Coretronic Intelligent Robotics (CIRC)”, available from: https://www.coretronic-robotics.com/tw/product2.
[2]“Bear Robotics”, available from: https://www.bearrobotics.ai/product.
[3]“Automated Guided Vehicle(AGV) - TECO”, available from: https://www.teco.com.tw/en/products/agv.
[4]H. van Hasselt, A. Guez, and D. Silver, 2016, “Deep reinforcement learning with double Q-Learning”, Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp. 2094–2100, February.
[5]Z. Wang, T. Schaul, M. Hessel, H. Van Hasselt, M. Lanctot, and N. De Freitas, 2016, “Dueling network architectures for deep reinforcement learning”, Proceedings of the 33rd International Conference on International Conference on Machine Learning, vol. 48, pp. 1995–2003, June.
[6]Y. Wang and T. Jiang, 2021, “Design and PLC Realization of Elevator Control Based on LOOK Algorithm”, Journal of Physics: Conference Series, vol. 1754, pp. 012082, February.
[7] “RMF”, available from: https://osrf.github.io/ros2multirobotbook/intro.html#robotics-middleware-framework-rmf.
[8]A. A. Abdulla, H. Liu, N. Stoll, and K. Thurow, 2016, “An automated elevator management and multi-floor estimation for indoor mobile robot transportation based on a pressure sensor”, 17th International Conference on Mechatronics - Mechatronika (ME), pp. 1–7, December.
[9]J.-G. Kang, S.-Y. An, and S.-Y. Oh, 2007, “Navigation strategy for the service robot in the elevator environment”, International Conference on Control, Automation and Systems, pp. 1092–1097, October.
[10]K. T. Islam, G. Mujtaba, R. G. Raj, and H. F. Nweke, 2017, “Elevator button and floor number recognition through hybrid image classification approach for navigation of service robot in buildings”, International Conference on Engineering Technology and Technopreneurship (ICE2T), pp. 1–4, September.
[11]X. Yuan, L. Buşoniu, and R. Babuška, 2008, “Reinforcement Learning for Elevator Control”, IFAC Proceedings Volumes, vol. 41, no. 2, pp. 2212–2217, July.
[12]M. Brand and D. Nikovski, 2004, “Risk-Averse Group Elevator Scheduling”, April.
[13]W. Liu, N. Liu, H. Sun, G. Xing, Y. Dong, and H. Chen, 2013, “Dispatching algorithm design for elevator group control system with Q-learning based on a recurrent neural network”, 25th Chinese Control and Decision Conference (CCDC), pp. 3397–3402, May.
[14]R. H. Crites and A. G. Barto, 1998, “Elevator Group Control Using Multiple Reinforcement Learning Agents”, vol. 33, no. 2, pp. 235–262, November.
[15]楊聖智, 2002, “運用基因演算法於控制電梯群體系統”, 國立臺灣師範大學-資訊教育研究所.
[16]A. Haj-Ali, N. K. Ahmed, T. Willke, J. Gonzalez, K. Asanovic, and I. Stoica, 2019, “A View on Deep Reinforcement Learning in System Optimization”, arXiv, Sep.
[17] S. Macenski, T. Foote, B. Gerkey, C. Lalancette, and W. Woodall, 2022, “Robot Operating System 2: Design, architecture, and uses in the wild”, Science Robotics, vol. 7, no. 66, p. eabm6074, May.
[18]“Gazebo”, available from: http://gazebosim.org/.
[19]“RViz”, available from: http://wiki.ros.org/rviz.
[20]“ROS2 topics”, available from: https://docs.ros.org/en/foxy/Tutorials/Beginner-CLI-Tools/Understanding-ROS2-Topics/Understanding-ROS2-Topics.html.
[21]“ROS2 services”, available from: https://docs.ros.org/en/foxy/Tutorials/Beginner-CLI-Tools/Understanding-ROS2-Services/Understanding-ROS2-Services.html.
[22]“ROS2 actions”, available from: https://docs.ros.org/en/foxy/Tutorials/Beginner-CLI-Tools/Understanding-ROS2-Actions/Understanding-ROS2-Actions.html.
[23]“Multicast”, available from: https://en.wikipedia.org/w/index.php?title=Multicast&oldid=1071824741.
[24] “RTPS”, available from: https://en.wikipedia.org/w/index.php?title=RTPS&oldid=933098532.
[25]“DDS API”, available from: https://fast-dds.docs.eprosima.com/en/latest/.
[26]Y. Maruyama, S. Kato, and T. Azumi, 2016, “Exploring the performance of ROS2”, Proceedings of the 13th International Conference on Embedded Software, pp. 1–10, Oct.
[27]“Open Robotics”, available from: https://www.openrobotics.org.
[28]“RMF Web”, available from: https://osrf.github.io/ros2multirobotbook/rmf-web.html.
[29]“Markov chain”, available from: https://en.wikipedia.org/w/index.php?title=Markov_chain&oldid=1087473341.
[30]C. J. C. H. Watkins and P. Dayan, 1992, “Q-learning”, Mach Learn, vol. 8, no. 3, pp. 279–292, May.
[31]V. Mnih et al., 2013, “Playing Atari with Deep Reinforcement Learning”, arXiv, Dec.
[32]H. Hasselt, 2010, “Double Q-learning”, Advances in Neural Information Processing Systems, vol. 23, June.
[33]“Pareto principle”, available from: https://en.wikipedia.org/w/index.php?title=Pareto_principle&oldid=1083488474