自駕車的崛起促使各國和公司積極開發並制定自駕車相關的規範，甚至有些國家已經開放特定區域的自駕車載客服務。然而，自駕車的開發過程中，訓練是不可或缺的一個階段，且需要大量的行車情境數據來支持，若完全在現實環境中進行訓練，將耗費大量時間和成本。為解決這一問題，許多自駕車開發團隊將訓練階段轉移到電腦上，通過模擬自駕車在現實環境中行駛的情境進行訓練。然而，模擬環境與現實環境存在著諸多差異，例如環境材質和光線細節等因素，這些差異會對自駕車在實際環境中的行車表現產生影響。
本論文提出了三種虛實整合方法，分別為sim2real、real2sim和Quasi-real2real，這些方法被應用於深度強化學習中的車道維持訓練，以改善在模擬環境中訓練的模型在現實環境中的性能。為了驗證這些方法在現實環境中的可行性和穩定性，本論文建立了一個與模擬環境相似的行駛環境進行測試。
最終，實驗結果得知Quasi-real2real與real2sim方法皆能有效縮小模擬與現實環境的差距，Quasi-real2real比不使用虛實整合方法的偏離次數少42%，而real2sim方法也有少39%偏離次數的表現。相反，sim2real方法的偏離次數則是還要多51%，代表無法有效縮小模擬與現實環境的差距。

The rise of autonomous vehicles has prompted countries and companies to actively develop and establish regulations related to self-driving cars, with some countries even allowing autonomous ride-hailing services in specific areas. To expedite the development process of autonomous vehicles, many teams have started shifting the training phase to computers, using simulations to train vehicles in real-world driving scenarios. However, there are differences between simulated and real environments that can impact the performance of autonomous vehicles in practical settings.
This paper proposes three virtual-to-real integration methods: sim2real, real2sim, and Quasi-real2real, for training lane-keeping tasks in deep reinforcement learning to improve the performance of models trained in simulated environments when deployed in real-world environments. To validate the feasibility and stability of these methods in real-world conditions, the paper establishes a driving environment similar to the simulated environment for testing purposes.
In conclusion, the experimental results indicate that both the Quasi-real2real and real2sim methods effectively reduce the gap between simulated and real-world environments. The Quasi-real2real method achieves a 42% reduction in deviation compared to not using any virtual-to-real integration method, while the real2sim method demonstrates a 39% reduction in deviation. Conversely, the sim2real method has a deviation rate that is 51% higher, indicating its inability to effectively bridge the gap between simulated and real-world environments.

摘要.............................................................................................................................................i
Abstract.......................................................................................................................................ii
誌謝...........................................................................................................................................iii
目錄...........................................................................................................................................iv
表目錄.......................................................................................................................................vi
圖目錄......................................................................................................................................vii
第一章 緒論..............................................................................................................................1
1.1 研究背景.....................................................................................................................1
1.2 研究動機與目的.........................................................................................................2
1.3 論文架構.....................................................................................................................3
第二章 文獻探討......................................................................................................................4
2.1 強化學習.....................................................................................................................4
2.1.1 探索與利用......................................................................................................4
2.1.2 Q-learning..........................................................................................................5
2.2 深度強化學習.............................................................................................................6
2.2.1 Deep Q-Network................................................................................................7
2.2.2 Dueling DQN.....................................................................................................8
2.3 Sim2Real ......................................................................................................................9
2.4 Real2Sim ....................................................................................................................10
2.5 CycleGAN .................................................................................................................. 11
2.6 Gym Duckietown........................................................................................................12
2.6.1 貝茲曲線........................................................................................................13
第三章 實驗方法與系統架構................................................................................................17
3.1 sim2real(虛轉實)........................................................................................................18
3.1.1 訓練流程(模擬環境)......................................................................................18
3.1.2 部署流程(現實環境)......................................................................................18
3.2 real2sim(實轉虛)........................................................................................................19
3.2.1 訓練流程(模擬環境)......................................................................................19
3.2.2 部署流程(現實環境)......................................................................................20
3.3 Quasi-real2real............................................................................................................21
3.3.1 訓練流程(模擬環境)......................................................................................23
3.3.2 部署流程(現實環境)......................................................................................23
3.4 車道維持訓練...........................................................................................................24
3.4.1 神經網路架構................................................................................................24
3.4.2 Dueling DQN 訓練流程..................................................................................25
3.4.3 訓練環境設置................................................................................................28
3.4.4 獎勵機制........................................................................................................28
3.5 CycleGAN 的訓練 .....................................................................................................29
3.6 車道維持測試...........................................................................................................30
3.6.1 實驗車輛........................................................................................................30
3.6.2 實驗場域........................................................................................................31
3.6.3 測試規則........................................................................................................33
3.7 特徵可視化...............................................................................................................35
第四章 實驗結果與討論........................................................................................................38
4.1 車道維持訓練結果...................................................................................................38
4.2 車道維持測試結果...................................................................................................44
4.3 特徵可視化比較.......................................................................................................48
4.4 訓練時間比較...........................................................................................................50
第五章 結論與未來研究........................................................................................................52
參考文獻..................................................................................................................................53
Extended Abstract.....................................................................................................................56

[1] F. Ding, K. Yu, Z. Gu, X. Li, and Y. Shi, "Perceptual enhancement for autonomous
vehicles: Restoring visually degraded images for context prediction via adversarial
training," IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 7, pp.
9430-9441, 2021.
[2] S. Gu, E. Holly, T. Lillicrap, and S. Levine, "Deep reinforcement learning for robotic
manipulation with asynchronous off-policy updates," in 2017 IEEE international
conference on robotics and automation (ICRA), 2017: IEEE, pp. 3389-3396.
[3] A. Gaidon, Q. Wang, Y. Cabon, and E. Vig, "Virtual worlds as proxy for multi-object
tracking analysis," in Proceedings of the IEEE conference on computer vision and
pattern recognition, 2016, pp. 4340-4349.
[4] G. C. Burdea and P. Coiffet, Virtual reality technology. John Wiley & Sons, 2003.
[5] R. T. Azuma, "A survey of augmented reality," Presence: teleoperators & virtual
environments, vol. 6, no. 4, pp. 355-385, 1997.
[6] R. Sutton and A. Barto, "Reinforcement Learning: An Introduction. in progress,
complete draft online," ed: MIT Press, November, 2017.
[7] C. J. C. H. Watkins, "Learning from delayed rewards," 1989.
[8] J. Kober, J. A. Bagnell, and J. Peters, "Reinforcement learning in robotics: A survey,"
The International Journal of Robotics Research, vol. 32, no. 11, pp. 1238-1274, 2013.
[9] V. Mnih et al., "Playing atari with deep reinforcement learning," arXiv preprint
arXiv:1312.5602, 2013.
[10] V. Mnih et al., "Human-level control through deep reinforcement learning," nature, vol.
518, no. 7540, pp. 529-533, 2015.
[11] H. Van Hasselt, A. Guez, and D. Silver, "Deep reinforcement learning with double qlearning," in Proceedings of the AAAI conference on artificial intelligence, 2016, vol.
30, no. 1.
[12] D. Silver et al., "Mastering the game of Go with deep neural networks and tree search,"
nature, vol. 529, no. 7587, pp. 484-489, 2016.
[13] A. A. Lydia and F. S. Francis, "Convolutional neural network with an optimized
backpropagation technique," in 2019 IEEE International Conference on System,
Computation, Automation and Networking (ICSCAN), 2019: IEEE, pp. 1-5.
[14] S. Albawi, T. A. Mohammed, and S. Al-Zawi, "Understanding of a convolutional neural
network," in 2017 international conference on engineering and technology (ICET),
2017: Ieee, pp. 1-6.
[15] GGWithRabitLIFE. "[機器學習 ML NOTE] Reinforcement Learning 強化學習
(DQN 原理)."
https://medium.com/%E9%9B%9E%E9%9B%9E%E8%88%87%E5%85%94%E5%8
54
5%94%E7%9A%84%E5%B7%A5%E7%A8%8B%E4%B8%96%E7%95%8C/%E6%
A9%9F%E5%99%A8%E5%AD%B8%E7%BF%92-ml-note-reinforcement-learning-
%E5%BC%B7%E5%8C%96%E5%AD%B8%E7%BF%92-dqn-
%E5%AF%A6%E4%BD%9Catari-game-7f9185f833b0 (accessed.
[16] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, "Dueling network
architectures for deep reinforcement learning," in International conference on machine
learning, 2016: PMLR, pp. 1995-2003.
[17] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired image-to-image translation using
cycle-consistent adversarial networks," in Proceedings of the IEEE international
conference on computer vision, 2017, pp. 2223-2232.
[18] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, "Domain
randomization for transferring deep neural networks from simulation to the real world,"
in 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS),
2017: IEEE, pp. 23-30.
[19] X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne, "Deepmimic: Example-guided
deep reinforcement learning of physics-based character skills," ACM Transactions On
Graphics (TOG), vol. 37, no. 4, pp. 1-14, 2018.
[20] T. P. Lillicrap et al., "Continuous control with deep reinforcement learning," arXiv
preprint arXiv:1509.02971, 2015.
[21] P. Chang and T. Padif, "Sim2real2sim: Bridging the gap between simulation and realworld in flexible object manipulation," in 2020 Fourth IEEE International Conference
on Robotic Computing (IRC), 2020: IEEE, pp. 56-62.
[22] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-image translation with
conditional adversarial networks," in Proceedings of the IEEE conference on computer
vision and pattern recognition, 2017, pp. 1125-1134.
[23] H.-y. Lee. "GAN Lecture 2 (2017): CycleGAN."
https://www.youtube.com/watch?v=9N_uOIPghuo (accessed.
[24] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz, "Multimodal unsupervised image-toimage translation," in Proceedings of the European conference on computer vision
(ECCV), 2018, pp. 172-189.
[25] Y. Taigman, A. Polyak, and L. Wolf, "Unsupervised cross-domain image generation,"
arXiv preprint arXiv:1611.02200, 2016.
[26] L. Paull et al., "Duckietown: an open, inexpensive and flexible platform for autonomy
education and research," in 2017 IEEE International Conference on Robotics and
Automation (ICRA), 2017: IEEE, pp. 1497-1504.
[27] M. a. G. Chevalier-Boisvert, Florian and Cao, Yanjun and Mehta, Bhairav and Paull,
Liam. "Duckietown Environments for OpenAI Gym." GitHub.
https://github.com/duckietown/gym-duckietown (accessed.
55
[28] ASUS. "RT-AC1200 V2｜無線路由器｜ASUS 台灣."
https://www.asus.com/tw/networking-iot-servers/wifi-routers/asus-wifi-routers/rtac1200-v2/techspec/ (accessed.
[29] M. Bojarski et al., "Explaining how a deep neural network trained with end-to-end
learning steers a car," arXiv preprint arXiv:1704.07911, 2017.