Reinforcement-Learning-Based Adaptive Optimal Flight Control with Output Feedback and Input Constraints

No AccessEngineering NotesReinforcement-Learning-Based Adaptive Optimal Flight Control with Output Feedback and Input ConstraintsBo Sun and Erik-Jan van KampenBo Sun https://orcid.org/0000-0002-5229-8545Delft University of Technology, 2629 HS Delft, The Netherlands and Erik-Jan van Kampen https://orcid.org/0000-0002-5593-4471Delft University of Technology, 2629 HS Delft, The NetherlandsPublished Online:8 Jun 2021https://doi.org/10.2514/1.G005715SectionsRead Now ToolsAdd to favoritesDownload citationTrack citations ShareShare onFacebookTwitterLinked InRedditEmail About References [1] Sun B. and van Kampen E.-J., "Incremental Model-Based Global Dual Heuristic Programming with Explicit Analytical Calculations Applied to Flight Control," Engineering Applications of Artificial Intelligence, Vol. 89, March 2020, Paper 103425. https://doi.org/10.1016/j.engappai.2019.103425 Google Scholar[2] Junell J., Mannucci T., Zhou Y. and van Kampen E.-J., "Self-Tuning Gains of a Quadrotor Using a Simple Model for Policy Gradient Reinforcement Learning," AIAA Guidance, Navigation, and Control Conference, AIAA Paper 2016-1387, 2016. https://doi.org/10.2514/6.2016-1387 LinkGoogle Scholar[3] Ferrari S. and Stengel R. F., "Online Adaptive Critic Flight Control," Journal of Guidance, Control, and Dynamics, Vol. 27, No. 5, 2004, pp. 777–786. https://doi.org/10.2514/1.12597 LinkGoogle Scholar[4] Zhou Y., van Kampen E.-J. and Chu Q. P., "Nonlinear Adaptive Flight Control Using Incremental Approximate Dynamic Programming and Output Feedback," Journal of Guidance, Control, and Dynamics, Vol. 40, No. 2, 2016, pp. 493–496. https://doi.org/10.2514/1.G001762 LinkGoogle Scholar[5] Heydari A. and Balakrishnan S. N., "Adaptive Critic-Based Solution to an Orbital Rendezvous Problem," Journal of Guidance, Control, and Dynamics, Vol. 37, No. 1, 2014, pp. 344–350. https://doi.org/10.2514/1.60553 LinkGoogle Scholar[6] Zhou Y., van Kampen E.-J. and Chu Q. P., "Incremental Approximate Dynamic Programming for Nonlinear Adaptive Tracking Control with Partial Observability," Journal of Guidance, Control, and Dynamics, Vol. 41, No. 12, 2018, pp. 2554–2567. https://doi.org/10.2514/1.G003472 LinkGoogle Scholar[7] Sutton R. S. and Barto A. G., Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA, 2018, pp. 1–4. Google Scholar[8] Sutton R. S., Barto A. G. and Williams R. J., "Reinforcement Learning Is Direct Adaptive Optimal Control," IEEE Control Systems Magazine, Vol. 12, No. 2, 1992, pp. 19–22. https://doi.org/10.1109/37.126844 CrossrefGoogle Scholar[9] Sun B. and van Kampen E.-J., "Launch Vehicle Discrete-Time Optimal Tracking Control Using Global Dual Heuristic Programming," 2020 IEEE Conference on Control Technology and Applications (CCTA), Inst. of Electrical and Electronics Engineers, New York, 2020, pp. 162–167. https://doi.org/10.1109/CCTA41146.2020.9206252 Google Scholar[10] Lewis F. L. and Vrabie D., "Reinforcement Learning and Adaptive Dynamic Programming for Feedback Control," IEEE Circuits and Systems Magazine, Vol. 9, No. 3, 2009, pp. 32–50. https://doi.org/10.1109/MCAS.2009.933854 CrossrefGoogle Scholar[11] Zhou Y., van Kampen E.-J. and Chu Q. P., "Incremental Model Based Online Dual Heuristic Programming for Nonlinear Adaptive Control," Control Engineering Practice, Vol. 73, April 2018, pp. 13–25. https://doi.org/10.1016/j.conengprac.2017.12.011 CrossrefGoogle Scholar[12] Heydari A. and Balakrishnan S. N., "Finite-Horizon Control-Constrained Nonlinear Optimal Control Using Single Network Adaptive Critics," IEEE Transactions on Neural Networks and Learning Systems, Vol. 24, No. 1, 2013, pp. 145–157. https://doi.org/10.1109/TNNLS.2012.2227339 CrossrefGoogle Scholar[13] Al-Tamimi A., Lewis F. L. and Abu-Khalaf M., "Discrete-Time Nonlinear HJB Solution Using Approximate Dynamic Programming: Convergence Proof," IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), Vol. 38, No. 4, 2008, pp. 943–949. https://doi.org/10.1109/TSMCB.2008.926614 CrossrefGoogle Scholar[14] Modares H. and Lewis F. L., "Optimal Tracking Control of Nonlinear Partially-Unknown Constrained-Input Systems Using Integral Reinforcement Learning," Automatica, Vol. 50, No. 7, 2014, pp. 1780–1792. https://doi.org/10.1016/j.automatica.2014.05.011 CrossrefGoogle Scholar[15] Kiumarsi B. and Lewis F. L., "Actor–Critic-Based Optimal Tracking for Partially Unknown Nonlinear Discrete-Time Systems," IEEE Transactions on Neural Networks and Learning Systems, Vol. 26, No. 1, 2015, pp. 140–151. https://doi.org/10.1109/TNNLS.2014.2358227 CrossrefGoogle Scholar[16] Van Kampen E.-J., Chu Q. P. and Mulder J., "Continuous Adaptive Critic Flight Control Aided with Approximated Plant Dynamics," AIAA Guidance, Navigation, and Control Conference and Exhibit, AIAA Paper 2006-6429, 2006. https://doi.org/10.2514/6.2006-6429 LinkGoogle Scholar[17] Sun B. and van Kampen E.-J., "Incremental Model-Based Heuristic Dynamic Programming with Output Feedback Applied to Aerospace System Identification and Control," 2020 IEEE Conference on Control Technology and Applications (CCTA), Inst. of Electrical and Electronics Engineers, New York, 2020, pp. 366–371. https://doi.org/10.1109/CCTA41146.2020.9206261 Google Scholar[18] Li H., Sun L., Tan W., Jia B. and Liu X., "Switching Flight Control for Incremental Model-Based Dual Heuristic Dynamic Programming," Journal of Guidance, Control, and Dynamics, Vol. 43, No. 7, 2020, pp. 1352–1358. https://doi.org/10.2514/1.G004519 LinkGoogle Scholar[19] Sun B. and van Kampen E.-J., "Incremental Model-Based Global Dual Heuristic Programming for Flight Control," IFAC-PapersOnLine, Vol. 52, No. 29, 2019, pp. 7–12. https://doi.org/10.1016/j.ifacol.2019.12.613 CrossrefGoogle Scholar[20] Lewis F. L. and Vamvoudakis K. G., "Reinforcement Learning for Partially Observable Dynamic Processes: Adaptive Dynamic Programming Using Measured Output Data," IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), Vol. 41, No. 1, 2010, pp. 14–25. https://doi.org/10.1109/TSMCB.2010.2043839 CrossrefGoogle Scholar[21] Tandale M. D. and Valasek J., "Adaptive Dynamic Inversion Control with Actuator Saturation Constraints Applied to Tracking Spacecraft Maneuvers," Journal of the Astronautical Sciences, Vol. 52, No. 4, 2004, pp. 517–530. https://doi.org/10.1007/BF03546415 CrossrefGoogle Scholar[22] Sonneveldt L., Chu Q. P. and Mulder J. A., "Nonlinear Flight Control Design Using Constrained Adaptive Backstepping," Journal of Guidance, Control, and Dynamics, Vol. 30, No. 2, 2007, pp. 322–336. https://doi.org/10.2514/1.25834 LinkGoogle Scholar[23] Modares H., Lewis F. L. and Naghibi-Sistani M.-B., "Integral Reinforcement Learning and Experience Replay for Adaptive Optimal Control of Partially-Unknown Constrained-Input Continuous-Time Systems," Automatica, Vol. 50, No. 1, 2014, pp. 193–202. https://doi.org/10.1016/j.automatica.2013.09.043 CrossrefGoogle Scholar[24] Sonneveldt L., Van Oort E. R., Chu Q. P. and Mulder J. A., "Nonlinear Adaptive Trajectory Control Applied to an F-16 Model," Journal of Guidance, Control, and Dynamics, Vol. 32, No. 1, 2009, pp. 25–39. https://doi.org/10.2514/1.38785 LinkGoogle Scholar[25] Nguyen L., Ogburn M., Gilbert W., Kibler K., Brown P. and Deal P., "Simulator Study of Stall/Post-Stall Characteristics of a Fighter Airplane with relaxed Longitudinal Static Stability," NASA TP 1538, 1979. Google Scholar Previous article Next article FiguresReferencesRelatedDetailsCited byObserver-based optimal control method combination with event-triggered strategy for hypersonic morphing vehicleAerospace Science and Technology, Vol. 136A Reinforcement Learning Method to Trajectory Design for Manned Lunar Mission via Reshaping Rewards31 January 2023Vision-Based Nonlinear Incremental Control for a Morphing Wing With Mechanical ImperfectionsIEEE Transactions on Aerospace and Electronic Systems, Vol. 58, No. 6Test and Evaluation of Reinforcement Learning via Robustness Testing and Explainable AI for High-Speed Aerospace VehiclesEvent-triggered intelligent critic control with input constraints applied to a nonlinear aeroelastic systemAerospace Science and Technology, Vol. 120Event-triggered constrained control using explainable global dual heuristic programming for nonlinear discrete-time systemsNeurocomputing, Vol. 468 What's Popular Volume 44, Number 9September 2021 CrossmarkInformationCopyright © 2021 by the American Institute of Aeronautics and Astronautics, Inc. All rights reserved. All requests for copying and permission to reprint should be submitted to CCC at www.copyright.com; employ the eISSN 1533-3884 to initiate your request. See also AIAA Rights and Permissions www.aiaa.org/randp. TopicsArtificial IntelligenceArtificial Neural NetworkComputing SystemComputing and InformaticsComputing, Information, and CommunicationControl TheoryData ScienceFeedback ControlGuidance, Navigation, and Control SystemsMachine LearningOptimal Control Theory KeywordsFull State FeedbackReinforcement LearningFlight ControlElevator DeflectionTracking ControlANNBang Bang ControlNonlinear SystemsAerospace SystemMonte Carlo SimulationAcknowledgmentThe authors would like to thank the Chinese Scholarship Council for financial support for B. Sun with the project reference number of 201806290007.PDF Received10 October 2020Accepted1 May 2021Published online8 June 2021