Hostname: page-component-78c5997874-8bhkd Total loading time: 0 Render date: 2024-11-10T07:25:42.277Z Has data issue: false hasContentIssue false
  • English
  • Français

Towards Augmented Microscopy with Reinforcement Learning-Enhanced Workflows

Published online by Cambridge University Press:  05 September 2022

Michael Xu ,
Abinash Kumar  and
James M. LeBeau Open the ORCID record for James M. LeBeau [Opens in a new window]
Michael Xu
Affiliation:
Department of Materials Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
Abinash Kumar
Affiliation:
Department of Materials Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
James M. LeBeau*
Affiliation:
Department of Materials Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
*
*Corresponding author: James M. LeBeau, E-mail: lebeau@mit.edu
Get access

Abstract

Here, we report a case study implementation of reinforcement learning (RL) to automate operations in the scanning transmission electron microscopy workflow. To do so, we design a virtual, prototypical RL environment to test and develop a network to autonomously align the electron beam position without prior knowledge. Using this simulator, we evaluate the impact of environment design and algorithm hyperparameters on alignment accuracy and learning convergence, showing robust convergence across a wide hyperparameter space. Additionally, we deploy a successful model on the microscope to validate the approach and demonstrate the value of designing appropriate virtual environments. Consistent with simulated results, the on-microscope RL model achieves convergence to the goal alignment after minimal training. Overall, the results highlight that by taking advantage of RL, microscope operations can be automated without the need for extensive algorithm design, taking another step toward augmenting electron microscopy with machine learning methods.

Keywords

automated microscopyreinforcement learningscanning transmission electron microscopy
Type
Software and Instrumentation
Information
Microscopy and Microanalysis , Volume 28 , Issue 6 , December 2022 , pp. 1952 - 1960
Check for updates
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press on behalf of the Microscopy Society of America

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Akers, S, Kautz, E, Trevino-Gavito, A, Olszta, M, Matthews, BE, Wang, L, Du, Y & Spurgeon, SR (2021). Rapid and flexible segmentation of electron microscopy data using few-shot machine learning. npj Comput Mater 7, 1–9.CrossRefGoogle Scholar
Amodei, D, Olah, C, Steinhardt, J, Christiano, P, Schulman, J & Mané, D (2016). Concrete problems in AI safety. arXiv:160606565.Google Scholar
Brockman, G, Cheung, V, Pettersson, L, Schneider, J, Schulman, J, Tang, J & Zaremba, W (2016). OpenAI Gym. arXiv:160601540.Google Scholar
Callaway, E (2020). Revolutionary cryo-EM is taking over structural biology. Nature 578, 201–201.CrossRefGoogle ScholarPubMed
Degrave, J, Felici, F, Buchli, J, Neunert, M, Tracey, B, Carpanese, F, Ewalds, T, Hafner, R, Abdolmaleki, A, de las Casas, D, Donner, C, Fritz, L, Galperti, C, Huber, A, Keeling, J, Tsimpoukelli, M, Kay, J, Merle, A, Moret, JM, Noury, S, Pesamosca, F, Pfau, D, Sauter, O, Sommariva, C, Coda, S, Duval, B, Fasoli, A, Kohli, P, Kavukcuoglu, K, Hassabis, D & Riedmiller, M (2022). Magnetic control of tokamak plasmas through deep reinforcement learning. Nature 602, 414–419.CrossRefGoogle ScholarPubMed
Ede, JM (2021 a). Adaptive partial scanning transmission electron microscopy with reinforcement learning. Mach Learn: Sci Technol 2, 045011.Google Scholar
Ede, JM (2021 b). Deep learning in electron microscopy. Mach Learn: Sci Technol 2, 011004.Google Scholar
Findlay, S & LeBeau, J (2013). Detector non-uniformity in scanning transmission electron microscopy. Ultramicroscopy 124, 52–60.CrossRefGoogle ScholarPubMed
Ge, M, Su, F, Zhao, Z & Su, D (2020). Deep learning analysis on microscopic imaging in materials science. Mater Today Nano 11, 100087.CrossRefGoogle Scholar
Gunawan, AA, Mkhoyan, KA, Wills, AW, Thomas, MG & Norris, DJ (2011). Imaging “invisible” dopant atoms in semiconductor nanocrystals. Nano Lett 11, 5553–5557.CrossRefGoogle ScholarPubMed
Haarnoja, T, Zhou, A, Abbeel, P & Levine, S (2018 a). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv:180101290.Google Scholar
Haarnoja, T, Zhou, A, Hartikainen, K, Tucker, G, Ha, S, Tan, J, Kumar, V, Zhu, H, Gupta, A, Abbeel, P & Levine, S (2018 b). Soft actor-critic algorithms and applications. arXiv:181205905.Google Scholar
Han, Y, Jang, J, Cha, E, Lee, J, Chung, H, Jeong, M, Kim, TG, Chae, BG, Kim, HG, Jun, S, Hwang, S, Lee, E & Ye, JC (2021). Deep learning STEM-EDX tomography of nanocrystals. Nat Mach Intell 3, 267–274.CrossRefGoogle Scholar
Harris, C (2022). Openchemistry/stempy: Stempy 3.0.0. Available at https://doi.org/10.5281/zenodo.6546416CrossRefGoogle Scholar
He, T, Wang, W, Shi, F, Yang, X, Li, X, Wu, J, Yin, Y & Jin, M (2021). Mastering the surface strain of platinum catalysts for efficient electrocatalysis. Nature 598, 76–81.CrossRefGoogle ScholarPubMed
Henderson, P, Islam, R, Bachman, P, Pineau, J, Precup, D & Meger, D (2018). Deep reinforcement learning that matters. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32.CrossRefGoogle Scholar
Islam, R, Henderson, P, Gomrokchi, M & Precup, D (2017). Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv:170804133.Google Scholar
Kober, J, Bagnell, JA & Peters, J (2013). Reinforcement learning in robotics: A survey. Int J Rob Res 32, 1238–1274.CrossRefGoogle Scholar
Kumar, A, Baker, JN, Bowes, PC, Cabral, MJ, Zhang, S, Dickey, EC, Irving, DL & LeBeau, JM (2021). Atomic-resolution electron microscopy of nanoscale local structure in lead-based relaxor ferroelectrics. Nat Mater 20, 62–67.CrossRefGoogle ScholarPubMed
LeBeau, J, Kumar, A & Hauwiller, M (2020). A universal scripting engine for transmission electron microscopy. Microsc Microanal 26, 2958–2959.CrossRefGoogle Scholar
LeBeau, JM & Stemmer, S (2008). Experimental quantification of annular dark field images in scanning transmission electron microscopy. Ultramicroscopy 108, 1653–1658.CrossRefGoogle ScholarPubMed
Lee, W, Nam, HS, Kim, YG, Kim, YJ, Lee, JH & Yoo, H (2021). Robust autofocusing for scanning electron microscopy based on a dual deep learning network. Sci Rep 11, 20933.CrossRefGoogle ScholarPubMed
Li, QJ, Sheng, H & Ma, E (2019). Strengthening in multi-principal element alloys with local chemical order roughened dislocation pathways. Nat Commun 10, 3563.CrossRefGoogle ScholarPubMed
Lillicrap, TP, Hunt, JJ, Pritzel, A, Heess, N, Erez, T, Tassa, Y, Silver, D & Wierstra, D (2015). Continuous control with deep reinforcement learning. arXiv:150902971.Google Scholar
Lin, R, Zhang, R, Wang, C, Yang, XQ & Xin, HL (2021). TEMImageNet training library and AtomSegNet deep-learning models for high-precision atom segmentation, localization, denoising, and deblurring of atomic-resolution images. Sci Rep 11, 5386.CrossRefGoogle ScholarPubMed
Lynnerup, NA, Nolling, L, Hasle, R & Hallam, J (2019). A survey on reproducibility by evaluating deep reinforcement learning algorithms on real-world robots. arXiv:190903772.Google Scholar
Mastronarde, DN (2005). Automated electron microscope tomography using robust prediction of specimen movements. J Struct Biol 152, 36–51.CrossRefGoogle ScholarPubMed
McKenzie, M & McDonnell, MD (2019). Degradation of performance in reinforcement learning with state measurement uncertainty. In 2019 Military Communications and Information Systems Conference (MilCIS), pp. 1–5.CrossRefGoogle Scholar
Navalón, S & García, H (2016). Nanoparticles for catalysis. Nanomaterials 6, 123.CrossRefGoogle ScholarPubMed
Nion Swift (2022). Nion Swift. Available at https://github.com/nion-software/nionswift (retrieved May 22, 2022).Google Scholar
Olszta, M, Hopkins, D, Fiedler, KR, Oostrom, M, Akers, S & Spurgeon, SR (2021). An automated scanning transmission electron microscope guided by sparse data analytics. arXiv:210914772.Google Scholar
PyJEM (2022). PyJEM. Available at https://github.com/PyJEM/PyJEM (retrieved May 22, 2022).Google Scholar
Raffin, A, Hill, A, Ernestus, M, Gleave, A, Kanervisto, A & Dormann, N (2019). Stable baselines3. Available at https://github.com/DLR-RM/stable-baselines3 (retrieved May 22, 2022).Google Scholar
Ragone, M, Saray, MT, Long, L, Shahbazian-Yassar, R, Mashayek, F & Yurkiv, V (2022). Deep learning for mapping element distribution of high-entropy alloys in scanning transmission electron microscopy images. Comput Mater Sci 201, 110905.CrossRefGoogle Scholar
Roccapriore, KM, Kalinin, SV & Ziatdinov, M (2021). Physics discovery in nanoplasmonic systems via autonomous experiments in scanning transmission electron microscopy. arXiv:210803290.Google Scholar
Silver, D, Huang, A, Maddison, CJ, Guez, A, Sifre, L, van den Driessche, G, Schrittwieser, J, Antonoglou, I, Panneershelvam, V, Lanctot, M, Dieleman, S, Grewe, D, Nham, J, Kalchbrenner, N, Sutskever, I, Lillicrap, T, Leach, M, Kavukcuoglu, K, Graepel, T & Hassabis, D (2016). Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489.CrossRefGoogle ScholarPubMed
Spurgeon, SR, Ophus, C, Jones, L, Petford-Long, A, Kalinin, SV, Olszta, MJ, Dunin-Borkowski, RE, Salmon, N, Hattar, K, Yang, WCD, Sharma, R, Du, Y, Chiaramonti, A, Zheng, H, Buck, EC, Kovarik, L, Penn, RL, Li, D, Zhang, X, Murayama, M & Taheri, ML (2021). Towards data-driven next-generation transmission electron microscopy. Nat Mater 20, 274–279.CrossRefGoogle ScholarPubMed
Sutton, RS & Barto, AG (2018). Reinforcement Learning: An Introduction, 2nd ed. Adaptive Computation and Machine Learning Series, Cambridge, MA: The MIT Press.Google Scholar
Tegunov, D & Cramer, P (2019). Real-time cryo-electron microscopy data preprocessing with warp. Nat Methods 16, 1146–1152.CrossRefGoogle ScholarPubMed
Thermo Fisher Scientific (2022). EPU 2 EM Software. Available at https://www.thermofisher.com/us/en/home/electron-microscopy/products/software-em-3d-vis/epu-software.htmlGoogle Scholar
Uusimaeki, T, Wagner, T, Lipinski, HG & Kaegi, R (2019). AutoEM: A software for automated acquisition and analysis of nanoparticles. J Nanopart Res 21, 122.CrossRefGoogle Scholar
Vasudevan, RK, Ghosh, A, Ziatdinov, M & Kalinin, SV (2021). Exploring electron beam induced atomic assembly via reinforcement learning in a molecular dynamics environment. arXiv:210411635.Google Scholar
Vinyals, O, Babuschkin, I, Czarnecki, WM, Mathieu, M, Dudzik, A, Chung, J, Choi, DH, Powell, R, Ewalds, T, Georgiev, P, Oh, J, Horgan, D, Kroiss, M, Danihelka, I, Huang, A, Sifre, L, Cai, T, Agapiou, JP, Jaderberg, M, Vezhnevets, AS, Leblond, R, Pohlen, T, Dalibard, V, Budden, D, Sulsky, Y, Molloy, J, Paine, TL, Gulcehre, C, Wang, Z, Pfaff, T, Wu, Y, Ring, R, Yogatama, D, Wünsch, D, McKinney, K, Smith, O, Schaul, T, Lillicrap, T, Kavukcuoglu, K, Hassabis, D, Apps, C & Silver, D (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354.CrossRefGoogle ScholarPubMed
Yang, S, Choi, W, Cho, BW, AgyapongFordjour, FO, Park, S, Yun, SJ, Kim, H, Han, Y, Lee, YH, Kim, KK & Kim, Y (2021). Deep learning assisted quantification of atomic dopants and defects in 2D materials. Adv Sci 8, 2101099.CrossRefGoogle ScholarPubMed
Yin, W, Brittain, D, Borseth, J, Scott, ME, Williams, D, Perkins, J, Own, CS, Murfitt, M, Torres, RM, Kapner, D, Mahalingam, G, Bleckert, A, Castelli, D, Reid, D, Lee, WCA, Graham, BJ, Takeno, M, Bumbarger, DJ, Farrell, C, Reid, RC & da Costa, NM (2020). A petascale automated imaging pipeline for mapping neuronal circuits with high throughput transmission electron microscopy. Nat Commun 11, 4949.CrossRefGoogle ScholarPubMed
Ziatdinov, M, Dyck, O, Maksov, A, Li, X, Sang, X, Xiao, K, Unocic, RR, Vasudevan, R, Jesse, S & Kalinin, SV (2017). Deep learning of atomically resolved scanning transmission electron microscopy images: Chemical identification and tracking local transformations. ACS Nano 11, 12742–12752.CrossRefGoogle ScholarPubMed
Ziatdinov, M, Ghosh, A, Wong, T & Kalinin, SV (2021 a). AtomAI: A deep learning framework for analysis of image and spectroscopy data in (scanning) transmission electron microscopy and beyond. arXiv:210507485.Google Scholar
Ziatdinov, M, Jesse, S, Sumpter, BG, Kalinin, SV & Dyck, O (2021 b). Tracking atomic structure evolution during directed electron beam induced Si atom motion in graphene via deep machine learning. Nanotechnology 32, 035703.Google Scholar

Xu et al. supplementary material

Xu et al. supplementary material
Download Xu et al. supplementary material(Video)
Video 16.7 MB