Hostname: page-component-745bb68f8f-5r2nc Total loading time: 0 Render date: 2025-01-13T13:34:54.323Z Has data issue: false hasContentIssue false
  • English
  • Français

Reinforcement-learning-based actuator selection method for active flow control

Published online by Cambridge University Press:  12 January 2023

Romain Paris Open the ORCID record for Romain Paris [Opens in a new window] ,
Samir Beneddine Open the ORCID record for Samir Beneddine [Opens in a new window]  and
Julien Dandois Open the ORCID record for Julien Dandois [Opens in a new window]
Romain Paris*
Affiliation:
DAAA, ONERA, Université Paris Saclay, F-92190 Meudon, France
Samir Beneddine
Affiliation:
DAAA, ONERA, Université Paris Saclay, F-92190 Meudon, France
Julien Dandois
Affiliation:
DAAA, ONERA, Université Paris Saclay, F-92190 Meudon, France
*
†Email address for correspondence: romain.paris@onera.fr
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper addresses the issue of actuator selection for active flow control by proposing a novel method built on top of a reinforcement learning agent. Starting from a pre-trained agent using numerous actuators, the algorithm estimates the impact of a potential actuator removal on the value function, indicating the agent's performance. It is applied to two test cases, the one-dimensional Kuramoto–Sivashinsky equation and a laminar bidimensional flow around an airfoil at $Re=1000$ for different angles of attack ranging from $12^{\circ }$ to $20^{\circ }$, to demonstrate its capabilities and limits. The proposed actuator-sparsification method relies on a sequential elimination of the least relevant action components, starting from a fully developed layout. The relevancy of each component is evaluated using metrics based on the value function. Results show that, while still being limited by this intrinsic elimination paradigm (i.e. the sequential elimination), actuator patterns and obtained policies demonstrate relevant performances and allow us to draw an accurate approximation of the Pareto front of performances versus actuator budget.

JFM classification

Flow Control: Drag reduction Flow Control: Instability control Mathematical Foundations: Machine learning
Type
JFM Papers
Information
Journal of Fluid Mechanics , Volume 955 , 25 January 2023 , A8
Check for updates
Copyright
© The Author(s), 2023. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Amitay, M. & Glezer, A. 2002 Role of actuation frequency in controlled flow reattachment over a stalled airfoil. AIAA J. 40 (2), 209–216.CrossRefGoogle Scholar
Beintema, G., Corbetta, A., Biferale, L. & Toschi, F. 2020 Controlling Rayleigh–Bénard convection via reinforcement learning. J. Turbul. 21 (9–10), 585–605.CrossRefGoogle Scholar
Belus, V., Rabault, J., Viquerat, J., Che, Z., Hachem, E. & Reglade, U. 2019 Exploiting locality and translational invariance to design effective deep reinforcement learning control of the 1-dimensional unstable falling liquid film. AIP Adv. 9 (12), 125014.CrossRefGoogle Scholar
Bhatnagar, S., Afshar, Y., Pan, S., Duraisamy, K. & Kaushik, S. 2019 Prediction of aerodynamic flow fields using convolutional neural networks. Comput. Mech. 64 (2), 525–545.CrossRefGoogle Scholar
Bhattacharjee, D., Hemati, M., Klose, B. & Jacobs, G. 2018 Optimal actuator selection for airfoil separation control. AIAA Paper 18-3692.CrossRefGoogle Scholar
Bruneau, C.-H. & Mortazavi, I. 2008 Numerical modelling and passive flow control using porous media. Comput. Fluids 37 (5), 488–498.CrossRefGoogle Scholar
Brunton, S.L. & Noack, B.R. 2015 Closed-loop turbulence control: progress and challenges. Appl. Mech. Rev. 67 (5), 050801.CrossRefGoogle Scholar
Brunton, S.L., Noack, B.R. & Koumoutsakos, P. 2020 Machine learning for fluid mechanics. Annu. Rev. Fluid Mech. 52, 477–508.CrossRefGoogle Scholar
Bucci, M.A., Semeraro, O., Allauzen, A., Cordier, L. & Mathelin, L. 2022 Nonlinear optimal control using deep reinforcement learning. In IUTAM Laminar-Turbulent Transition, pp. 279–290. Springer.CrossRefGoogle Scholar
Bucci, M.A., Semeraro, O., Allauzen, A., Wisniewski, G., Cordier, L. & Mathelin, L. 2019 Control of chaotic systems by deep reinforcement learning. Proc. R. Soc. A 475 (2231), 20190351.CrossRefGoogle ScholarPubMed
Chomaz, J.-M. 2005 Global instabilities in spatially developing flows: non-normality and nonlinearity. Annu. Rev. Fluid Mech. 37, 357–392.CrossRefGoogle Scholar
Cohen, K., Siegel, S. & McLaughlin, T. 2006 A heuristic approach to effective sensor placement for modeling of a cylinder wake. Comput. Fluids 35 (1), 103–120.CrossRefGoogle Scholar
Dandois, J., Mary, I. & Brion, V. 2018 Large-eddy simulation of laminar transonic buffet. J. Fluid Mech. 850, 156–178.CrossRefGoogle Scholar
Djeumou, F., Neary, C., Goubault, E., Putot, S. & Topcu, U. 2022 Neural networks with physics-informed architectures and constraints for dynamical systems modeling. In Learning for Dynamics and Control Conference (ed. N. Lawrence & M. Reid), pp. 263–277. PMLR.Google Scholar
Duraisamy, K., Iaccarino, G. & Xiao, H. 2019 Turbulence modeling in the age of data. Annu. Rev. Fluid Mech. 51, 357–377.CrossRefGoogle Scholar
Edwards, J.R. & Liou, M.-S. 1998 Low-diffusion flux-splitting methods for flows at all speeds. AIAA J. 36 (9), 1610–1617.Google Scholar
Evans, H.B., Hamed, A.M., Gorumlu, S., Doosttalab, A., Aksak, B., Chamorro, L.P. & Castillo, L. 2018 Engineered bio-inspired coating for passive flow control. Proc. Natl Acad. Sci. USA 115 (6), 1210–1214.CrossRefGoogle Scholar
Fukami, K., Hasegawa, K., Nakamura, T., Morimoto, M. & Fukagata, K. 2021 Model order reduction with neural networks: application to laminar and turbulent flows. SN Comput. Sci. 2 (6), 1–16.CrossRefGoogle Scholar
Garnier, P., Viquerat, J., Rabault, J., Larcher, A., Kuhnle, A. & Hachem, E. 2021 A review on deep reinforcement learning for fluid mechanics. Comput. Fluids 225, 104973.CrossRefGoogle Scholar
Ghosh, S., Das, N., Das, I. & Maulik, U. 2019 Understanding deep learning techniques for image segmentation. ACM Comput. Surv. 52 (4), 1–35.CrossRefGoogle Scholar
Giannetti, F. & Luchini, P. 2007 Structural sensitivity of the first instability of the cylinder wake. J. Fluid Mech. 581, 167–197.CrossRefGoogle Scholar
Gupta, R. & Ansell, P.J. 2019 Unsteady flow physics of airfoil dynamic stall. AIAA J. 57 (1), 165–175.Google Scholar
Hämäläinen, P., Babadi, A., Ma, X. & Lehtinen, J. 2020 PPO-CMA: Proximal policy optimization with covariance matrix adaptation. In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing, pp. 1–6. IEEE.CrossRefGoogle Scholar
Hasegawa, K., Fukami, K., Murata, T. & Fukagata, K. 2020 Machine-learning-based reduced-order modeling for unsteady flows around bluff bodies of various shapes. Theor. Comput. Fluid Dyn. 34 (4), 367–383.Google Scholar
Hui, X., Bai, J., Wang, H. & Zhang, Y. 2020 Fast pressure distribution prediction of airfoils using deep learning. Aerosp. Sci. Technol. 105, 105949.CrossRefGoogle Scholar
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P. & Levine, S. 2021 How to train your robot with deep reinforcement learning: lessons we have learned. Intl J. Rob. Res. 40 (4–5), 698–721.CrossRefGoogle Scholar
Jin, B., Illingworth, S.J. & Sandberg, R.D. 2022 Optimal sensor and actuator placement for feedback control of vortex shedding. J. Fluid Mech. 932, A2.CrossRefGoogle Scholar
Joubert, G., Le Pape, A., Heine, B. & Huberson, S. 2013 Vortical interactions behind deployable vortex generator for airfoil static stall control. AIAA J. 51 (1), 240–252.Google Scholar
Kingma, D.P. & Ba, J. 2015 Adam: a method for stochastic optimization. In 3rd International Conference on Learning Representations, Conference Track Proceedings.Google Scholar
Kneer, S., Sayadi, T., Sipp, D., Schmid, P. & Rigas, G. 2021 Symmetry-aware autoencoders: s-PCA and s-nlPCA. arXiv:2111.02893.Google Scholar
Kochkov, D., Smith, J.A., Alieva, A., Wang, Q., Brenner, M.P. & Hoyer, S. 2021 Machine learning-accelerated computational fluid dynamics. Proc. Natl Acad. Sci. USA 118 (21), e2101784118.CrossRefGoogle ScholarPubMed
Koizumi, H., Tsutsumi, S. & Shima, E. 2018 Feedback control of Kármán vortex shedding from a cylinder using deep reinforcement learning. AIAA Paper 18-3691.CrossRefGoogle Scholar
Lee, K. & Carlberg, K.T. 2020 Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. J. Comput. Phys. 404, 108973.CrossRefGoogle Scholar
Li, J. & Zhang, M. 2022 Reinforcement-learning-based control of confined cylinder wakes with stability analyses. J. Fluid Mech. 932, A44.CrossRefGoogle Scholar
Louizos, C., Welling, M. & Kingma, D.P. 2018 Learning sparse neural networks through $l_0$ regularization. In Sixth International Conference on Learning Representations.Google Scholar
Luhar, M., Sharma, A.S. & McKeon, B.J. 2014 Opposition control within the resolvent analysis framework. J. Fluid Mech. 749, 597–626.CrossRefGoogle Scholar
Lusch, B., Kutz, J.N. & Brunton, S.L. 2018 Deep learning for universal linear embeddings of nonlinear dynamics. Nat. Commun. 9 (1), 4950.CrossRefGoogle ScholarPubMed
Manohar, K., Kutz, J.N. & Brunton, S.L. 2021 Optimal sensor and actuator selection using balanced model reduction. IEEE Trans. Automat. Contr. 67 (4), 2108–2115.Google Scholar
Mao, Y., Zhong, S. & Yin, H. 2022 Active flow control using deep reinforcement learning with time delays in Markov decision process and autoregressive policy. Phys. Fluids 34 (5), 053602.CrossRefGoogle Scholar
Mary, I. 1999 Méthode de newton approchée pour le calcul d’écoulements instationnaires comportant des zones à très faibles nombres de mach. PhD thesis, Paris 11.Google Scholar
Milano, M. & Koumoutsakos, P. 2002 Neural network modeling for near wall turbulent flow. J. Comput. Phys. 182 (1), 1–26.CrossRefGoogle Scholar
Mohan, A.T. & Gaitonde, D.V. 2018 A deep learning based approach to reduced order modeling for turbulent flow control using LSTM neural networks. arXiv:1804.09269.Google Scholar
Natarajan, M., Freund, J.B. & Bodony, D.J. 2016 Actuator selection and placement for localized feedback flow control. J. Fluid Mech. 809, 775–792.CrossRefGoogle Scholar
Oehler, S.F. & Illingworth, S.J. 2018 Sensor and actuator placement trade-offs for a linear model of spatially developing flows. J. Fluid Mech. 854, 34–55.CrossRefGoogle Scholar
Otter, D.W., Medina, J.R. & Kalita, J.K. 2020 A survey of the usages of deep learning for natural language processing. IEEE Trans. Neural Netw. Learn. Syst. 32 (2), 604–624.CrossRefGoogle Scholar
Paris, R., Beneddine, S. & Dandois, J. 2021 Robust flow control and optimal sensor placement using deep reinforcement learning. J. Fluid Mech. 913, A25.CrossRefGoogle Scholar
Pichi, F., Ballarin, F., Rozza, G. & Hesthaven, J.S. 2021 An artificial neural network approach to bifurcating phenomena in computational fluid dynamics. arXiv:2109.10765.Google Scholar
Rabault, J., Kuchta, M., Jensen, A., Réglade, U. & Cerardi, N. 2019 Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. J. Fluid Mech. 865, 281–302.CrossRefGoogle Scholar
Rabault, J. & Kuhnle, A. 2019 Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach. Phys. Fluids 31 (9), 094105.CrossRefGoogle Scholar
Rabault, J., Ren, F., Zhang, W., Tang, H. & Xu, H. 2020 Deep reinforcement learning in fluid mechanics: a promising method for both active flow control and shape optimization. J. Hydrodyn. 32 (2), 234–246.CrossRefGoogle Scholar
Ren, F., Rabault, J. & Tang, H. 2021 Applying deep reinforcement learning to active flow control in weakly turbulent conditions. Phys. Fluids 33 (3), 037121.CrossRefGoogle Scholar
Rogers, J. 2000 A parallel approach to optimum actuator selection with a genetic algorithm. AIAA Paper 2000–4484.Google Scholar
Roshko, A. 1953 On the development of turbulent wakes from vortex streets, report 1191. Tech. Rep. 2913. California Institue of Technology.Google Scholar
Sashittal, P. & Bodony, D.J. 2021 Data-driven sensor placement for fluid flows. Theor. Comput. Fluid Dyn. 35 (5), 709–729.CrossRefGoogle Scholar
Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. 2017 Proximal policy optimization algorithms. arXiv:1707.06347.Google Scholar
Seidel, J., Fagley, C. & McLaughlin, T. 2018 Feedback flow control: a heuristic approach. AIAA J. 56 (10), 3825–3834.CrossRefGoogle Scholar
Seifert, A., Bachar, T., Koss, D., Shepshelovich, M. & Wygnanski, I. 1993 Oscillatory blowing: a tool to delay boundary-layer separation. AIAA J. 31 (11), 2052–2060.CrossRefGoogle Scholar
Seifert, A., Darabi, A. & Wyganski, I. 1996 Delay of airfoil stall by periodic excitation. J. Aircraft 33 (4), 691–698.CrossRefGoogle Scholar
Seifert, A. & Pack, L.G. 1999 Oscillatory control of separation at high Reynolds numbers. AIAA J. 37 (9), 1062–1071.Google Scholar
Sekar, V., Zhang, M., Shu, C. & Khoo, B.C. 2019 Inverse design of airfoil using a deep convolutional neural network. AIAA J. 57 (3), 993–1003.CrossRefGoogle Scholar
Seshagiri, A., Cooper, E. & Traub, L.W. 2009 Effects of vortex generators on an airfoil at low Reynolds numbers. J. Aircraft 46 (1), 116–122.CrossRefGoogle Scholar
Shimomura, S., Ogawa, T., Sekimoto, S., Nonomura, T., Oyama, A., Fujii, K. & Nishida, H. 2017 Experimental analysis of closed-loop flow control around airfoil using DBD plasma actuator. In ASME 2017 Fluids Engineering Division Summer Meeting (ed. ASME), p. V01CT22A004. American Society of Mechanical Engineers.Google Scholar
Shimomura, S., Sekimoto, S., Oyama, A., Fujii, K. & Nishida, H. 2020 Closed-loop flow separation control using the deep Q network over airfoil. AIAA J. 58 (10), 4260–4270.CrossRefGoogle Scholar
Sivashinsky, G.I. 1980 On flame propagation under conditions of stoichiometry. SIAM J. Appl. Maths 39 (1), 67–82.CrossRefGoogle Scholar
Vinuesa, R. & Brunton, S.L. 2021 The potential of machine learning to enhance computational fluid dynamics. arXiv:2110.02085.Google Scholar
Vona, M. & Lauga, E. 2021 Stabilizing viscous extensional flows using reinforcement learning. Phys. Rev. E 104 (5), 055108.CrossRefGoogle ScholarPubMed
Wang, J.-X., Wu, J.-L. & Xiao, H. 2017 Physics-informed machine learning approach for reconstructing Reynolds stress modeling discrepancies based on DNS data. Phys. Rev. Fluids 2 (3), 034603.CrossRefGoogle Scholar
Wang, Y.-Z., Mei, Y.-F., Aubry, N., Chen, Z., Wu, P. & Wu, W.-T. 2022 Deep reinforcement learning based synthetic jet control on disturbed flow over airfoil. Phys. Fluids 34 (3), 033606.CrossRefGoogle Scholar
Wang, Z., Xiao, D., Fang, F., Govindan, R., Pain, C.C. & Guo, Y. 2018 Model identification of reduced order fluid dynamics systems using deep learning. Intl J. Numer. Meth. Fluids 86 (4), 255–268.CrossRefGoogle Scholar
Willcox, K. 2006 Unsteady flow sensing and estimation via the gappy proper orthogonal decomposition. Comput. Fluids 35 (2), 208–226.CrossRefGoogle Scholar
Wong, J.C., Ooi, C., Gupta, A. & Ong, Y.-S. 2022 Learning in sinusoidal spaces with physics-informed neural networks. IEEE Trans. Artif. Intell.Google Scholar
Wu, J.-Z., Lu, X.-Y., Denny, A.G., Fan, M. & Wu, J.-M. 1998 Post-stall flow control on an airfoil by local unsteady forcing. J. Fluid Mech. 371, 21–58.CrossRefGoogle Scholar
Yao, H., Sun, Y. & Hemati, M.S. 2022 Feedback control of transitional shear flows: sensor selection for performance recovery. Theor. Comput. Fluid Dyn. 36 (4), 597–626.CrossRefGoogle Scholar
Yeh, C.-A. & Taira, K. 2019 Resolvent-analysis-based design of airfoil separation control. J. Fluid Mech. 867, 572–610.CrossRefGoogle Scholar
Zaman, K.B.M.Q., McKinzie, D.J. & Rumsey, C.L. 1989 A natural low-frequency oscillation of the flow over an airfoil near stalling conditions. J. Fluid Mech. 202, 403–442.CrossRefGoogle Scholar
Zhang, T., Wang, R., Wang, Y. & Wang, S. 2021 Locomotion control of a hybrid propulsion biomimetic underwater vehicle via deep reinforcement learning. In 2021 IEEE International Conference on Real-time Computing and Robotics (ed. M.K. O'Malley), pp. 211–216. IEEE.CrossRefGoogle Scholar
Zhang, Y., Sung, W.J. & Mavris, D.N. 2018 Application of convolutional neural network to predict airfoil lift coefficient. AIAA Paper 2018-1903.CrossRefGoogle Scholar