Deep Reinforcement Learning Approach for Integrated Updraft Mapping and Exploitation

No AccessEngineering NotesDeep Reinforcement Learning Approach for Integrated Updraft Mapping and ExploitationStefan Notter, Christian Gall, Gregor Müller, Aamir Ahmad and Walter FichterStefan Notter https://orcid.org/0000-0002-7632-3119University of Stuttgart, 70569 Stuttgart, Germany*Research Associate, Institute of Flight Mechanics and Controls.Search for more papers by this author, Christian GallUniversity of Stuttgart, 70569 Stuttgart, Germany*Research Associate, Institute of Flight Mechanics and Controls.Search for more papers by this author, Gregor MüllerUniversity of Stuttgart, 70569 Stuttgart, Germany†Graduate Student.Search for more papers by this author, Aamir AhmadUniversity of Stuttgart, 70569 Stuttgart, Germany‡Tenure-Track Professor, Institute of Flight Mechanics and Controls.Search for more papers by this author and Walter FichterUniversity of Stuttgart, 70569 Stuttgart, Germany§Professor, Institute of Flight Mechanics and Controls. Associate Fellow AIAA.Search for more papers by this authorPublished Online:25 Aug 2023https://doi.org/10.2514/1.G007572SectionsRead Now ToolsAdd to favoritesDownload citationTrack citations ShareShare onFacebookTwitterLinked InRedditEmail About References [1] Allen M., “Autonomous Soaring for Improved Endurance of a Small Uninhabited Air Vehicle,” 43rd AIAA Aerospace Sciences Meeting and Exhibit, AIAA Paper 2005-1025, 2005. https://doi.org/10.2514/6.2005-1025 LinkGoogle Scholar[2] Allen M. and Lin V., “Guidance and Control of an Autonomous Soaring Vehicle with Flight Test Results,” 45th AIAA Aerospace Sciences Meeting and Exhibit, AIAA Paper 2005-0867, 2007. https://doi.org/10.2514/6.2007-867 LinkGoogle Scholar[3] Edwards D., “Implementation Details and Flight Test Results of an Autonomous Soaring Controller,” AIAA Guidance, Navigation and Control Conference and Exhibit, AIAA Paper 2005-7244, 2008. https://doi.org/10.2514/6.2008-7244 LinkGoogle Scholar[4] Andersson K., Kaminer I., Dobrokhodov V. and Cichella V., “Thermal Centering Control for Autonomous Soaring; Stability Analysis and Flight Test Results,” Journal of Guidance, Control, and Dynamics, Vol. 35, No. 3, 2012, pp. 963–975. https://doi.org/10.2514/1.51691 LinkGoogle Scholar[5] Bird J. and Langelaan J., “Spline Mapping to Maximize Energy Exploitation of Non-Uniform Thermals,” Technical Soaring, Vol. 37, No. 3, 2013, pp. 38–44. Google Scholar[6] Depenbusch N. and Langelaan J., “Coordinated Mapping and Exploration for Autonomous Soaring,” Infotech@ Aerospace 2011, AIAA Paper 2011-1436, 2011. https://doi.org/10.2514/6.2011-1436 LinkGoogle Scholar[7] Cheng K. and Langelaan J. W., “Guided Exploration for Coordinated Autonomous Soaring Flight,” AIAA Guidance, Navigation, and Control Conference, AIAA Paper 2014-0969, 2014. https://doi.org/10.2514/6.2014-0969 LinkGoogle Scholar[8] Depenbusch N. T., Bird J. J. and Langelaan J. W., “The AutoSOAR Autonomous Soaring Aircraft, Part 1: Autonomy Algorithms,” Journal of Field Robotics, Vol. 35, No. 6, 2018, pp. 868–889. https://doi.org/10.1002/rob.21782 CrossrefGoogle Scholar[9] Depenbusch N. T., Bird J. J. and Langelaan J. W., “The AutoSOAR Autonomous Soaring Aircraft, Part 2: Hardware Implementation and Flight Results,” Journal of Field Robotics, Vol. 35, No. 4, 2018, pp. 435–458. https://doi.org/10.1002/rob.21747 CrossrefGoogle Scholar[10] Lawrance N. R. J. and Sukkarieh S., “Path Planning for Autonomous Soaring Flight in Dynamic Wind Fields,” 2011 IEEE International Conference on Robotics and Automation, IEEE, New York, 2011, pp. 2499–2505. https://doi.org/10.1109/ICRA.2011.5979966 Google Scholar[11] Reddy G., Celani A., Sejnowski T. J. and Vergassola M., “Learning to Soar in Turbulent Environments,” Proceedings of the National Academy of Sciences, Vol. 113, No. 33, 2016, pp. E4877–E4884. https://doi.org/10.1073/pnas.1606075113 CrossrefGoogle Scholar[12] Reddy G., Wong-Ng J., Celani A., Sejnowski T. J. and Vergassola M., “Glider Soaring via Reinforcement Learning in the Field,” Nature, Vol. 562, No. 7726, 2018, pp. 236–239. https://doi.org/10.1038/s41586-018-0533-0 CrossrefGoogle Scholar[13] Edwards D. J. and Silberberg L. M., “Autonomous Soaring: The Montague Cross-Country Challenge,” Journal of Aircraft, Vol. 47, No. 5, 2010, pp. 1763–1769. https://doi.org/10.2514/1.C000287 LinkGoogle Scholar[14] Kahn A. D., “Atmospheric Thermal Location Estimation,” Journal of Guidance, Control, and Dynamics, Vol. 40, No. 9, 2017, pp. 2363–2369. https://doi.org/10.2514/1.g002782 LinkGoogle Scholar[15] Oettershagen P., Stastny T., Hinzmann T., Rudin K., Mantel T., Melzer A., Wawrzacz B., Hitz G. and Siegwart R., “Robotic Technologies for Solar-Powered UAVs: Fully Autonomous Updraft-Aware Aerial Sensing for Multiday Search-and-Rescue Missions,” Journal of Field Robotics, Vol. 35, No. 4, 2018, pp. 612–640. https://doi.org/10.1002/rob.21765 CrossrefGoogle Scholar[16] Guilliard I., Rogahn R., Piavi J. and Kolobov A., “Autonomous Thermalling as a Partially Observable Markov Decision Process,” Robotics: Science and Systems XIV, Robotics: Science and Systems Foundation, 2018, https://roboticsproceedings.org/rss14/p68.html. https://doi.org/10.15607/rss.2018.xiv.068 Google Scholar[17] Notter S., Schrapel P., Groß P. and Fichter W., “Estimation of Multiple Thermal Updrafts Using a Particle Filter Approach,” 2018 AIAA Guidance, Navigation, and Control Conference, AIAA Paper 2018-1854, 2018. https://doi.org/10.2514/6.2018-1854 LinkGoogle Scholar[18] Notter S., Groß P., Schrapel P. and Fichter W., “Multiple Thermal Updraft Estimation and Observability Analysis,” Journal of Guidance, Control, and Dynamics, Vol. 43, No. 3, 2020, pp. 490–503. https://doi.org/10.2514/1.G004205 LinkGoogle Scholar[19] Notter S., Schimpf F. and Fichter W., “Hierarchical Reinforcement Learning Approach Towards Autonomous Cross-Country Soaring,” AIAA Scitech 2021 Forum, AIAA Paper 2021-2010, 2021. https://doi.org/10.2514/6.2021-2010 LinkGoogle Scholar[20] Notter S., Schimpf F., Müller G. and Fichter W., “Hierarchical Reinforcement Learning Approach for Autonomous Cross-Country Soaring,” Journal of Guidance, Control, and Dynamics, Vol. 46, No. 1, 2023, pp. 114–126. https://doi.org/10.2514/1.G006746 LinkGoogle Scholar[21] Schimpf F., Notter S., Groß P. and Fichter W., “Multi-Agent Reinforcement Learning for Thermalling in Updrafts,” AIAA Scitech 2021 Forum, AIAA Paper 2021-0864, 2021. https://doi.org/10.2514/6.2021-0864 LinkGoogle Scholar[22] Groß P., Notter S. and Fichter W., “Estimating Total Energy Compensated Climb Rates from Position Trajectories,” AIAA Scitech 2019 Forum, AIAA Paper 2019-0828, 2019. https://doi.org/10.2514/6.2019-0828 LinkGoogle Scholar[23] Sutton R. S. and Barto A. G., Reinforcement Learning: An Introduction, 2nd ed., MIT Press, Cambridge, MA, 2018, pp. 1–22, Chap. 1, Introduction. Google Scholar[24] Sutton R. S. and Barto A. G., Reinforcement Learning: An Introduction, 2nd ed., MIT Press, Cambridge, MA, 2018, pp. 321–338, Chap. 13, Policy Gradient Methods. Google Scholar[25] Schulman J., Levine S., Abbeel P., Jordan M. and Moritz P., “Trust Region Policy Optimization,” International Conference on Machine Learning, PMLR, 2015, pp. 1889–1897, https://proceedings.mlr.press/v37/schulman15.html. Google Scholar[26] Schulman J., Wolski F., Dhariwal P., Radford A. and Klimov O., “Proximal Policy Optimization Algorithms,” Computing Research Repository (CoRR), Vol. abs/1707.06347, 2017. Google Scholar[27] Brockman G., Cheung V., Pettersson L., Schneider J., Schulman J., Tang J. and Zaremba W., “OpenAI Gym,” 2016, https://arxiv.org/pdf/1606.01540.pdf. Google Scholar[28] Allen M. J., “Updraft Model for Development of Autonomous Soaring Uninhabited Air Vehicles,” 44th AIAA Aerospace Sciences Meeting and Exhibit, AIAA Paper 2006-1510, 2006. https://doi.org/10.2514/6.2006-1510 LinkGoogle Scholar[29] Paszke A., Gross S., Massa F., Lerer A., Bradbury J., Chanan G., Killeen T., Lin Z., Gimelshein N., Antiga L., Desmaison A., Kopf A., Yang E., DeVito Z., Raison M., Tejani A., Chilamkurthy S., Steiner B., Fang L., Bai J. and Chintala S., “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” Advances in Neural Information Processing Systems 32, Curran Associates, Inc., Red Hook, NY, 2019, pp. 8024–8035. Google Scholar[30] Meier L., Tanskanen P., Fraundorfer F. and Pollefeys M., “PIXHAWK: A System for Autonomous Flight Using Onboard Computer Vision,” 2011 IEEE International Conference on Robotics and Automation, IEEE, New York, 2011, pp. 2992–2997. https://doi.org/10.1109/ICRA.2011.5980229 Google Scholar Previous article Next article FiguresReferencesRelatedDetails What's Popular Volume 46, Number 10October 2023 CrossmarkInformationCopyright © 2023 by Institute of Flight Mechanics and Controls, University of Stuttgart. Published by the American Institute of Aeronautics and Astronautics, Inc., with permission. All requests for copying and permission to reprint should be submitted to CCC at www.copyright.com; employ the eISSN 1533-3884 to initiate your request. See also AIAA Rights and Permissions www.aiaa.org/randp. TopicsAeronauticsAircraft ControlAircraft Flight Control SystemAircraft Operations and TechnologyAircraft Stability and ControlAircraftsArtificial IntelligenceArtificial Neural NetworkAviationAviation SafetyComputing and InformaticsComputing SystemComputing, Information, and CommunicationData ScienceFlight TestGuidance, Navigation, and Control SystemsMachine LearningRoboticsUnmanned Aerial Vehicle KeywordsReinforcement LearningArtificial Neural NetworkFlight TestingGuidance, Navigation, and Control SystemsRoboticsIntelligent Flight Control SystemAerospace EngineeringAutonomous SoaringAutonomous Aerial VehicleAcknowledgmentThe topic presented has mainly been investigated within the project “Decision Making for Environmental Energy Exploitation with Small Aircraft,” funded by the Cyber Valley Research Fund (CyVy-RF-2021-21). The financial support is gratefully acknowledged.PDF Received1 March 2023Accepted14 July 2023Published online25 August 2023