An Acquisition Parameter Study for Machine-Learning-Enabled Electron Backscatter Diffraction

Abstract

Methods within the domain of artificial intelligence are gaining traction for solving a range of materials science objectives, notably the use of deep neural networks for computer vision for the analysis of electron diffraction patterns. An important component of deploying these models is an understanding of the performance as experimental diffraction conditions are varied. This knowledge can inspire confidence in the classifications over a range of operating conditions and identify where performance is degraded. Elucidating the relative impact of each parameter will suggest the most important parameters to vary during the collection of future training data. Knowing which data collection efforts to prioritize is of concern given the time required to collect or simulate vast libraries of diffraction patterns for a wide variety of materials without considering varying any parameters. In this work, five parameters, frame averaging, detector tilt, sample-to-detector distance, accelerating voltage, and pattern resolution, essential to electron diffraction are individually varied during the collection of electron backscatter diffraction patterns to explore the effect on the classifications produced by a deep neural network trained from diffraction patterns captured using a fixed set of parameters. The model is shown to be resilient to nearly all the individual changes examined here.

Introduction

The fields of materials development and analysis have recently begun to explore the possibility of applying data-driven strategies and artificial intelligence (AI) for accelerating or automating a variety of tasks (Ong et al., 2013; O'Mara et al., 2016; Dagdelen et al., 2017; Ward et al., 2018; Ziletti et al., 2018; Avery et al., 2019; DeCost et al., 2019; Oviedo et al., 2019; Tshitoyan et al., 2019; Dunn et al., 2020; Kaufmann & Vecchio, 2020; Kaufmann et al., 2020a, 2020d; Stan et al., 2020; Chen et al., 2021). Computer vision is a subset of AI with the goal of training computers to understand the visual world and potentially act on that information (Zhu et al., 2020a, 2020b). Deep learning algorithms enable many computer vision applications and are of particular interest owing to their excellent performance without significant feature engineering (LeCun et al., 2015). While it can be difficult to precisely determine how and why these “black-box algorithms” are capable of performing these tasks, these methods can automate routine tasks, improve upon existing solutions, or provide understanding (Adadi & Berrada, 2018; Holm, 2019). While deep learning provides significant opportunities for the advancement of materials science, robust application of these tools requires an understanding of the conditions under which optimal performance is achieved. Unlike most cases involving natural images (Deng et al., 2010), scientific images are less likely to be collected utilizing a wide range of parameters. For instance, micrographs may be captured with only 1 or 2 magnifications, while photographs of animals at different distances and from different angles are typically abundant (Deng et al., 2010). Moreover, the knowledge of the conditions for which a model fails can guide the future collection of labeled training data to improve subsequent versions. However, in the context of electron diffraction, there likely exists some cases (e.g., no frame averaging) where reduced quality or lack of sufficient information in the images may limit performance regardless of training data available for a given technique.

Convolutional neural networks (CNNs) are the current standard deep learning models for processing image data (LeCun et al., 2015). Before a CNN can be applied to a given task, it must learn to assign importance to various aspects of the image that maximize the network's differentiation capabilities. In doing so, deep learning methods glean intricate functions of the inputs that are sensitive to minute details, yet ignore irrelevant information such as backgrounds (LeCun et al., 2015). However, it should be noted that it is only with careful design of a training set and rigorous validation that practitioners can be confident that the model has truly learned relevant information, is robust to new conditions, and has not found an unscientific approach to solving the problem [such as learning the presence of a ruler means a lesion is more likely cancerous (Esteva et al., 2017)] (Zech et al., 2018; Riley, 2019). The application of these tools to image-based tasks in materials science has proved to be useful for classification (Modarres et al., 2017; Ziletti et al., 2018; Foden et al., 2019a; Kaufmann et al., 2020a), segmentation (DeCost et al., 2019; Stan et al., 2020), and other objectives (Xie & Grossman, 2018; de Haan et al., 2019). Examples of techniques where interest in developing artificial intelligence agents for image-based tasks include optical microscopy (DeCost & Holm, 2015; DeCost et al., 2019), scanning transmission electron microscopy (STEM) (Laanait et al., 2019; Roberts et al., 2019), transmission electron microscopy (TEM) (Spurgeon et al., 2020), and electron backscatter diffraction (EBSD) (Shen et al., 2019; Ding et al., 2020; Kaufmann et al., 2020a, 2020b, 2020c). These efforts are motivated by accelerating data generation rates and the traditional need for tedious or arduous analysis of the data by well-trained individuals with sufficient knowledge of the material domain. Thus, it is important that these tools can be applied robustly as imaging parameters are varied to prevent increasing, instead of alleviating, researcher workloads. While different diffraction techniques often use distinct terminology, parameters such as accelerating voltage or detector distance are common among them and have similar effects on the collected diffraction patterns. Several of these parameters play a role in the amount of time required to collect each diffraction pattern and therefore complete the analysis of a sample. In this work, the EBSD technique is used owing to the relatively high rate of data collection and ease of changing each parameter.

EBSD is a scanning electron microscope (SEM)-based method that involves the capture of 2D diffraction patterns produced from an incident electron beam scattering, diffracting, and escaping from a well-polished “bulk” sample (Schwartz et al., 2009). Despite the vast amount of information in the patterns (Vecchio & Williams, 1987, 1988; Schwartz et al., 2009), conventional EBSD has primarily focused on determining three-dimensional orientation (Schwartz et al., 2009; Thomsen et al., 2013; Tong et al., 2019; Zhu et al., 2020b). Furthermore, the technique typically relies on a user-defined phase list and Hough-based indexing (Lassen, 1994). Hough-based indexing generally allows for phase differentiation of sufficiently distinct crystal structures (Britton et al., 2010; Foden et al., 2019b; Hielscher et al., 2019), but the process remains susceptible to structural misclassification (McLaren & Reddy, 2008; Chen & Thomson, 2010; Karthikeyan et al., 2013). Improvements to phase differentiation have been proposed and developed including dictionary indexing (Chen et al., 2015; Ram et al., 2017; Ram & De Graef, 2018; Singh et al., 2018) and spherical indexing (Day, 2008; Lenthe et al., 2019; Zhu et al., 2019), although each still requires a user to pre-select phases and further requires simulating the Kikuchi sphere for each selected phase. Recently, the EBSD community has begun to investigate the use of CNNs for indexing, phase differentiation, and determining components of the crystal structure (Foden et al., 2019a; Ding et al., 2020; Kaufmann et al., 2020a, 2020b, 2020c). It is a goal of several of these efforts that the onus of phase selection and/or structure determination can be at least partially lessened on the user (Kaufmann et al., 2020a, 2020b). However, to date, there has not been a systematic study of CNN performance when the EBSD patterns (EBSPs) are collected using different experimental geometry than was used during collection or simulation of the training data. The knowledge of how these changes to the diffraction patterns influence proper pattern identification is paramount to widespread adoption of these machine-learning-based techniques.

This work seeks to develop an understanding of model performance as several of the most common EBSD operating conditions are varied. The specific parameters are frame averaging, detector tilt, sample-to-detector distance, accelerating voltage, and pattern resolution. With regard to parameters that directly affect the time to collect each pattern and therefore complete a map, such as frame averaging and pattern resolution, the ability to collect the data more rapidly without a significant reduction in performance is of interest. With respect to parameters such as detector tilt, it is important to determine if the model is susceptible to minor or major changes in the training conditions. Parameters such as detector distance and accelerating voltage can cause much more dramatic changes to the EBSPs, and it is, therefore, necessary to assess their influence. The CNN model tested in this work was trained to classify EBSPs to one of six space groups using patterns collected from a fixed EBSD setup (Kaufmann et al., 2020d). The effect of changing these parameters is tested using new EBSPs collected from one material from each space group and a dual-phase 2205 duplex steel for visual demonstration. Each time one parameter is varied, the EBSPs are re-collected, and the CNN used to reassess the proper space group identification. Ultimately, the model is found to retain a high classification accuracy even with significant changes to the diffraction conditions and therefore the EBSPs.

Materials and Methods

Materials

Eighteen different single-phase materials, comprising six of the ten space groups within the $( {4/m\;\bar{3}\;2/m} )$ point group were selected for training the space group classification CNN. The space groups are $Pm\bar{3}m\;( {221} )$, $Pm\bar{3}n\;( {223} )$, $Fm\bar{3}m\;( {225} )$, $Fd\bar{3}m\;( {227} )$, $\;Im\bar{3}m\;( {229} )$, and $\;Ia\bar{3}d\;( {230} )$. Suitable samples for the remaining four space groups could not be obtained. The materials were [221: FeAl, NiAl, Ni₃Al, Fe₃Ni], [223: Cr₃Si, Mo₃Si], [225: Ni, Al, NbC, TaC, TiC], [227: Si, Ge], [229: W, Ta, Fe], and [230: Al₄CoNi₂, and Al₄Ni₃]. The collected diffraction patterns from these materials were of low texture, typically less than two times random in any direction. Refer to Kaufmann et al. (2020d) for the distributions of orientation, band contrast, and mean angular deviation for these samples.

A dual-phase material with phases that are easily differentiable by Hough-based EBSD was used to visually demonstrate and compare the classification accuracy of the model as parameters vary. A longitudinal cross-section of cold-worked 2205 duplex steel was mounted, ground, and polished to 0.05 μm using colloidal silica. Looking at the longitudinal cross-section, cold-worked 2205 duplex stainless steel has a ferrite matrix (space group 229) and elongated austenite islands (space group 225) (Momeni et al., 2012).

Electron Backscatter Diffraction Pattern Collection

EBSPs were collected in a Thermo Scientific (formerly FEI) Apreo scanning electron microscope (SEM) equipped with an Oxford Symmetry EBSD detector.

All EBSPs utilized in training were collected with the following fixed geometry. The EBSD detector was utilized in high-resolution (1,244 × 1,024) mode, the working distance was 18.1 ± 0.1 mm, the sample-to-detector distance was 19.1 mm, and the detector arm tilt set to 13.7° above horizontal. The imaging parameters for the training set EBSPs were 20 kV accelerating voltage, 51 nA beam current, and 30-pattern averaging. This fixed geometry and set of imaging parameters will be referred to as the “default” operating conditions. Refer to Figure 1 for an annotated image of the diffraction setup in the SEM.

Fig. 1. EBSD setup and variable parameters. An annotated view of the EBSD setup. Parameters that vary in this study are listed below the equipment label. The sample-to-detector distance (DD) and detector tilt (DT) are further detailed by arrows describing their function. Detector tilt is defined as the angle between the detector arm and horizontal.

Diffraction patterns from six single-phase materials (Ni₃Al, Cr₃Si, TiC, Si, Fe, and Al₄CoNi₂) and the dual-phase 2205 duplex steel were collected separately from the training data to determine a baseline accuracy for the model when using the default diffraction geometry. Approximately 3,000 individual patterns were collected from a large area of each sample (i.e., low magnification) to capture as many unique orientations as possible over the fixed region. After collecting data using the default diffraction geometry, each one of the parameters was systematically varied one at a time and the same ~3,000 patterns re-collected from the same location.

The parameters were varied as follows. Frame averaging was set to 1, 5, 10, 20, or 30. The software detector tilt below zero ranged from 1 to 5 in steps of 1. These values correspond to the detector arm being 14.2°, 14.0°, 13.7°, 13.5°, and 13.3° above the horizontal plane. The software detector insertion distance ranged from 156 to 164 mm in steps of 2 mm. These values correspond to sample-to-detector distances of 24.3, 21.8, 19.1, 16.8, and 14.3 mm. The sample-to-detector distances were calculated following the methods outlined in Jackson et al. (2019). Accelerating voltage options were 10, 20, or 30 kV. The pattern resolution options were 156 × 128 (low), 622 × 512 (medium), or 1,244 × 1,024 (high). The working distance was held constant, since moving the sample up or down while at 70° sample tilt would change the location on the sample. The beam current also remained fixed and the exposure time adjusted accordingly by the Oxford Aztec software to offset the increase or decrease in signal resulting from a varied parameter (e.g., detector distance). Table A.1 summarizes the EBSD pattern acquisition rate for each of the varied parameters compared with the “default” conditions.

Convolutional Neural Network

The Xception CNN architecture (Chollet, 2017) was selected for fitting the model. The selection of this network was based on Xception or derivatives of Xception being used previously in the EBSD community (Ding et al., 2020; Kaufmann et al., 2020a, 2020b, 2020c). Refer to Figure 5 in Xception: Deep Learning with Depthwise Separable Convolutions (Chollet, 2017) for a complete description of the Xception architecture. Further details about the training process can be found in Kaufmann et al. (2020d). The CNNs were implemented with TensorFlow (Abadi et al., 2016) and Keras (Chollet, 2015).

Diffraction Pattern Classification

During pattern collection, Hough-based indexing was performed with only three options: ferrite, austenite, or non-indexed. After collecting high-resolution EBSPs from each material, all patterns collected were exported as tiff images. Diffraction patterns were evaluated in a random order by the trained CNN model without further information to assess the model as it would be applied in practice. The output classification of each diffraction pattern was recorded and saved in a (.csv) file. These csv files were utilized to calculate the normalized accuracy of the model for each trial. In the case of 2205 duplex steel, the predictions were converted to a space group map using the plotting tools in MATLAB R2018B following the methods established in Kaufmann et al. (2020c).

Results

Equipment Setup and Hough-Based Results

The choice of operating parameters for EBSD are not fixed but are instead valid over a range of values depending on the manufacturer's calibration. The specific parameters defined as “default” parameters in this work are those from which all EBSPs in the training set were collected [refer to the section “Electron backscatter diffraction pattern collection” or prior work by Kaufmann et al. (2020d)]. The equipment setup for EBSD is shown and labeled in Figure 1. The detector distance from the sample and detector tilt are further detailed using arrows that describe their geometric role. EBSPs are collected from a fixed region of the specimen each time one parameter is varied.

Figure 2 highlights the need for a reliable tool to assist an operator with symmetry determination using a section of 2205 duplex steel. In the case where the phases are known in advance (Fig. 2a), Hough-based EBSD is shown to produce a high-fidelity phase map of the austenite islands (blue) in the ferrite matrix (yellow). However, in cases where the phases are unknown, there exists ample opportunity for misclassification in the Hough-based method even when using what could be considered ideal acquisition parameters. If the six space groups used in the CNN study were selected in duplicate, one copy with lattice parameters matching the austenite's lattice parameters and symmetry and the other copy matching the ferrite's, the results can be strikingly different (Fig. 2b). In fact, not a single pattern is indexed to either of the two correct phases. Furthermore, two of the three space groups selected, $Pm\bar{3}n$ (orange) and $Fd\bar{3}m$ (lime green), are not present in the sample. Thus, a robust classifier of EBSD patterns would be a useful tool for assisting with selecting the proper phases or alerting users to potential errors in selected phase lists. Figure 2c demonstrates how a CNN-based classifier could be used for such a purpose. The CNN has predicted the space group of each high-quality EBSP that Hough-based EBSD used for generating Figures 2a and 2b, yet the CNN-derived phase map is a much higher fidelity mapping of Figure 2a than Figure 2b, albeit without the lattice parameter information at this stage of the CNN model. These results are significant since Figure 2b demonstrates a marked failure wherein Hough-based EBSD cannot distinguish between the two correct answers plus ten phases with the same lattice parameters but different space group symmetry. Given the potential the CNN-based method has for improving the EBSD process, this work sets out to study how reliable the CNN's classifications are as new data captured under different diffraction conditions is presented to it.

Fig. 2. Hough-based EBSD results with different phase lists. A comparison of the Hough-based EBSD method is performed where (a) only the two correct phases are provided as options and (b) 12 options are made available based on the six space groups used in this work and the two lattice parameters found in the duplex steel sample. The CNN-derived space group map (c) details the classifier's predictions for the same EBSD patterns used in parts (a) and (b). The data for each map are collected using the ideal parameters defined as “default” in this work. It is worth noting that the CNN-derived map is derived from having the solution for each EBSP classified from among the six cubic space groups in the trained model.

Effects on the EBSD Patterns

It is important to understand the effect of varying the operating conditions of the SEM and EBSD detector on the electron diffraction patterns. Figure 3 visually details the effects by displaying the same diffraction pattern observed under different operating conditions. The larger versions of these images are contained in Figures A.1–A.5. Increasing the number of frames averaged for each diffraction pattern increases the signal-to-noise ratio and results in better resolution of finer details in the diffraction patterns. Changing the detector tilt changes the relative position of the image with respect to the interaction volume of the sample, referred to as the pattern center in EBSD (Schwartz et al., 2009; Basinger et al., 2011; Zhu et al., 2019). For tilt angles that are far from the ideal conditions, the top or bottom edge of the EBSP may display blurring. The most significant blurring is observed when using a tilt of 13.3°. Otherwise, the changes are observed to be limited to small differences in the region of the Kikuchi sphere captured. Decreasing the sample-to-detector distance, and thus moving closer to the sample, results in the capture of significantly more solid angle. Since the EBSD detector is capturing a gnomonic projection, the increase in the solid angle means a greater area of the Kikuchi sphere is observed. This likely must be balanced with the ability to resolve the finer details and eventual blurring of the pattern edges. On the other hand, moving the detector further away captures less of the Kikuchi sphere but “magnifies” the finer details. The diffraction elements also appear slightly blurred at the furthest distance away from the sample. Changes to the SEM accelerating voltage affect the EBSPs by altering the wavelength of the incoming electrons. A decrease in the accelerating voltage increases the electron wavelength and therefore causes the Kikuchi bands to appear wider and vice versa. This results in more diffraction information from the same region of the Kikuchi sphere condensed within the viewing window for higher accelerating voltages. Note that it does not have the same effect as changing the detector distance. Lastly, the Oxford Symmetry detector can capture patterns at one of three resolutions: 156 × 128, 622 × 512, or 1,244 × 1,024. These are described as low, medium, and high resolution, respectively, in this work. For each of the different imaging conditions shown, approximately 3,000 diffraction patterns are collected from the same region of each sample and the CNNs performance analyzed.

Fig. 3. Impact of operating conditions on the EBSD patterns. The diffraction pattern for a point on the sample is displayed for different imaging conditions. The effect of the number of frames averaged, the tilt of the EBSD detector, the sample-to-detector distance for the EBSD detector, the SEM accelerating voltage, and the resolution at which patterns are captured are each visually described.

Frame Averaging

Figure 4 presents a visual overview of the pattern classification results for each frame averaging condition studied using a sample of 2205 duplex stainless steel, a material with a ferrite matrix (space group 229; yellow) and austenite islands (space group 225; blue) (Momeni et al., 2012). Similar space group maps for each of the parameters studied are shown in the Appendix (Figs. A.6–A.10). The electron image (Fig. 4a) and Hough-based phase map (Fig. 4b) are provided as the ground truth, since these phases can be differentiated by Hough with relative ease, assuming that the operator selects the correct phases at the start (i.e., user knows the phases). Without any frame averaging, the EBSPs are lacking much of the available details and essentially all the EBSPs from this sample are misidentified (Fig. 4c). Increasing the frame averaging to five improves the result (Fig. 4d), but it is not until ten frames are averaged that a reasonable number of classifications appear to be correct (Fig. 4e) when compared with the Hough-based map (Fig. 4b). Figures 4f and 4g each show sequential improvements over Figure 4e as nearly all of the patterns become correctly classified to the correct space group (i.e., matching the Hough-based phase map). As seen in the plot (Fig. 4h), the number of patterns classified to each space group begins to improve with ten frames averaged. While the plot shows the relative number of patterns classified to each space group compared with Hough-based EBSD, the space group maps made from the CNN's predictions establish whether the individual EBSPs are correctly classified based on their correlation with the Hough-based phase map. Since each of the other materials studied in this work is known to be single-phase, normalized classification accuracy precisely describes the model performance.

Fig. 4. Visual overview of frame averaging on CNN classification for duplex steel. (a) Electron image of the region of dual-phase 2205 duplex steel. (b) Hough-based EBSD phase map of the fcc (225) austenite (blue) and bcc (229) ferrite (yellow). (c) Phase map generated from EBSD patterns collected with no frame averaging applied (i.e., one frame). (d) Phase map generated from EBSD patterns collected with five frame averaging applied. (e) Phase map generated from EBSD patterns collected with ten frame averaging applied. (f) Phase map generated from EBSD patterns collected with 20 frame averaging applied. (g) Phase map generated from EBSD patterns collected with 30 frame averaging applied. (h) Plot showing the fraction of patterns indexed to each space group as a function of frame averaging. Thirty frame averaging is the default parameter and is designated as such by the blue star for space group 225 and a yellow star for space group 229. Trend lines are fit with a third order polynomial. Scale bar = 25 μm. There are 3,848 diffraction patterns (pixels) in each phase map.

It is worth pointing out that the Hough-based result is considered the ground truth (or correct answer) because the phases in the sample are known, and the known answer is provided as input to the Hough-based solution. So in this context, the EBSD solution for phase ID must already be known for the Hough-based approach. The power of the machine learning approach being implemented here, and previously presented in Kaufmann et al. (2020d), is to perform symmetry identification, to the space group level, for samples for which the phases present are not known a priori (i.e., enable true phase ID by EBSD).

Figure 5 details the normalized accuracy of the CNN for each space group as the number of frames averaged is varied in subsequent collections of the same EBSPs. The default frame averaging is 30 patterns. It is observed that a high overall classification accuracy, compared with the accuracy obtained for the default parameter, is generally retained as low as 5–10 frames averaged. An exception to this is observed for space groups 221 and 229 which, along with space group 225, are the most difficult for the CNN to differentiate owing to the strong similarities between the fcc and L1₂ structures and bcc and B2 structures used in training the model (Kaufmann et al., 2020d).

Fig. 5. Effect of frame averaging on classification accuracy. The normalized classification accuracy of the trained CNN for each space group based on the number of patterns averaged during data collection. The space groups are (a) $Pm\bar{3}m$, (b) $Pm\bar{3}n$, (c) $Fm\bar{3}m$, (d) $Pd\bar{3}m$, (e) $Im\bar{3}m$, and (f) $Ia\bar{3}d$. The default number of frames averaged is 30.

Detector Tilt

Figure 6 details the effect of the detector tilt on the CNN's classification of the collected EBSPs. Note that the detector tilt is reported as the angle of the detector arm above horizontal. Referring to the EBSPs in Figure 3 and their enlarged counterparts shown in Figure A.2, detector tilts of 14.2–13.7° show ideal patterns with little or no blurring at the edges. The blurring becomes more apparent at the bottom of the EBSPs when a tilt of 13.5° is applied and increases in severity at a tilt of 13.3°. For most space groups in Figure 6, the model is found to be highly resilient, and it is only at or below a tilt of 13.5° that any notable changes in accuracy occur. Again, space groups 221 and 229 are the exception to this observation. Referring to the symmetry maps of the duplex steel, the fractions do not vary significantly across the individual maps (Fig. A.7) and the space group classification of each EBSP aligns well with the Hough-based phase map (ground truth).

Fig. 6. Effect of detector tilt on classification accuracy. The normalized classification accuracy of the trained CNN for each space group based on the tilt of the detector during data collection. The space groups are (a) $Pm\bar{3}m$, (b) $Pm\bar{3}n$, (c) $Fm\bar{3}m$, (d) $Pd\bar{3}m$, (e) $Im\bar{3}m$, and (f) $Ia\bar{3}d$. The detector tilts are reported as the detector arm angle above horizontal. The default value for detector tilt is 13.7°.

Detector Distance

Figure 7 presents the changes in the CNN's performance as the EBSD detector collects the diffraction patterns at different sample-to-detector distances. The number of misclassified pixels, however minute the difference is, is observed to scale with an increased absolute distance from the default condition of 19.1 mm. When the detector is much further away from the sample, less solid angle is captured, and the diffraction patterns become distorted (Fig. 3 and Fig. A.3). Moving closer to the sample increases the solid angle, but at the expense of the finer details. This likely contributes to the noticeably reduced performance for the closest possible setting (a sample-to-detector distance of 14.3 mm). Referring to the phase maps of the duplex steel (Fig. A.8), similar effects are observed. As the sample-to-detector distance gets further from the default condition, the CNN-derived maps increasingly differ from the Hough-based phase map.

Fig. 7. Effect of sample-to-detector distance on classification accuracy. The normalized classification accuracy of the trained CNN for each space group based on the sample-to-detector distance during data collection. The space groups are (a) $Pm\bar{3}m$, (b) $Pm\bar{3}n$, (c) $Fm\bar{3}m$, (d) $Pd\bar{3}m$, (e) $Im\bar{3}m$, and (f) $Ia\bar{3}d$. The default value for sample-to-detector distance is 19.1 mm.

Accelerating Voltage

The effect of changing the wavelength of the incoming electrons by modifying the accelerating voltage is explored in Figure 8 and Figure A.9. The primary effect of changing the wavelength of the incoming electrons is a change in the width of the observed Kikuchi bands (Fig. 3 and Fig. A.4). The resulting effect on the collected diffraction patterns is similar to changing the detector distance, but notably does not alter the amount of solid angle captured on the phosphor screen. Instead, changing the accelerating voltage effectively changes the magnification of the diffraction data within the same screen area. For example, decreasing to 10 kV causes the details of the diffraction pattern to increase in size, effectively making the data appear expanded. Note how the same zone axes are present for each accelerating voltage, but the distance between the zone axes and the width of the diffraction lines appears to change (Fig. A.4). These changes are observed to have appreciable effects on the classification performance of the CNN. At 10 kV, many of the patterns from each space group are misclassified (Fig. 8). On the other hand, increasing to 30 kV accelerating voltage yields reasonably good classification performance, perhaps because the Kikuchi bands are narrower and more information appears to be visible. The same effects on performance are visually evident in the CNN-derived space group maps of the duplex steel (Fig. A.9).

Fig. 8. Effect of accelerating voltage on CNN classification. The normalized classification accuracy of the trained CNN for each space group based on the SEM accelerating voltage. The space groups are (a) $Pm\bar{3}m$, (b) $Pm\bar{3}n$, (c) $Fm\bar{3}m$, (d) $Pd\bar{3}m$, (e) $Im\bar{3}m$, and (f) $Ia\bar{3}d$. An accelerating voltage of 20 kV is the default.

Pattern Resolution

Reducing the initial resolution of the collected diffraction patterns bins the information from neighboring pixels to accelerate the rate of collection. Figure 9 details the CNN's performance after collecting the diffraction patterns at each of the three available resolutions for the Oxford Symmetry EBSD detector. The CNN is capable of achieving a high degree of accuracy even when patterns are collected at the lowest resolution setting. Only space groups 221 and 229 are appreciably impacted owing to the strong similarities between the fcc and L1₂ structures and bcc and B2 structures used in training the CNN. Figure A.10 visually demonstrates the reliable performance of the CNN at each pattern resolution by creating structure maps for the 2205 duplex steel.

Fig. 9. Effect of pattern resolution on CNN classification. The normalized classification accuracy of the trained CNN for each space group based on the EBSD detector resolution. Available resolutions are 156 × 128 (low), 622 × 512 (medium), and 1,244 × 1,024 (high). The space groups are (a) $Pm\bar{3}m$, (b) $Pm\bar{3}n$, (c) $Fm\bar{3}m$, (d) $Pd\bar{3}m$, (e) $Im\bar{3}m$, and (f) $Ia\bar{3}d$. High resolution (1,244 × 1,024) is the default setting.

Discussion

The electron diffraction community has recently begun to consider artificial intelligence a necessary component of next-generation microscopy (Spurgeon et al., 2020). Much of this is driven by the rate at which modern microscopes can generate high-quality data (Goulden et al., 2018); as a result, human knowledge and experience will no longer be an efficient means of analysis. If AI-based tools are to be implemented effectively, it is necessary for the community to identify potential areas of fragility, for example, changing diffraction conditions, and assess the impact in order to further development and increase trust in these “black-box” models.

EBSD was selected as an ideal technique owing to the ease of varying each parameter and the rates of data collection achievable (Schwartz et al., 2009; Goulden et al., 2018). Furthermore, several recent studies have explored the use of CNNs applied to components of EBSD analysis (Jha et al., 2018; Shen et al., 2019; Ding et al., 2020; Kaufmann et al., 2020a, 2020b); however, these efforts have thus far only been tested with new data collected or simulated using identical geometry and diffraction conditions. Herein, a parametric analysis of five parameters commonly found among electron diffraction techniques was performed to examine the performance of a CNN trained to classify diffraction patterns. One material was selected per space group and new diffraction patterns were collected starting with diffraction conditions matching the training set and then the same patterns were recollected 16 more times after setting just one parameter to a value different from the default conditions. The same analysis was performed for a sample of 2205 duplex steel to demonstrate CNN-based symmetry mapping (Kaufmann et al., 2020c) a sample with each varied parameter. If the trained CNN is highly sensitive to the diffraction conditions, we would have expected to see large decreases in performance with the smallest changes. For example, by changing the sample-to-detector distance, the Kikuchi bands from the same material can appear wider or narrower and the distance between diffraction information (e.g., zone axes) appears to change. However, the CNN's classification accuracy is observed to be quite stable (i.e., small reductions in classification accuracy) in comparison to the results achieved with the default conditions, suggesting that the features detectors learned by the model are not biased to these characteristics. This was one of the intended goals of using multiple materials with different z-contrast and lattice parameters for the same space group in the training set (Kaufmann et al., 2020d), and this study indicates its effectiveness. Moreover, the CNN is also observed to be highly dependable after decreasing the signal-to-noise ratio of the captured diffraction pattern by reducing the frame averaging. Decreasing the number of frames averaged will allow for faster data collection, as much as six times faster if averaging 5 frames compared to 30, while maintaining a high degree of classification accuracy for most materials. In each of the studies within this work, the space groups most likely to be misclassified by the model were 221 $( {Pm\bar{3}m\;} )$ and 229 $( {Im\bar{3}m\;} )$. As previously mentioned, the misclassification of these patterns can be at least partially attributed to the strong similarities between diffraction patterns from the fcc $( {Fm\bar{3}m} )$ and L1₂ $( {Pm\bar{3}m} )$ structures and bcc $( {Im\bar{3}m} )$ and B2 $( {Pm\bar{3}m} )$ structures used in training the CNN (Kaufmann et al., 2020d). Inclusion of more diverse data for these space groups may help alleviate this concern in addition to being a practical advancement toward commercial adoption. While the model's dependability and trustworthiness with respect to equipment parameters has now been evaluated for phase differentiation and identification problems, it is still important to test this hypothesis for EBSD orientation indexing CNNs (Jha et al., 2018; Shen et al., 2019; Ding et al., 2020). A much larger study should also be performed to investigate the effects of co-varying operating parameters; however, we expect that similar tolerances will be observed based on the reliability observed under diffraction conditions far from the training data. This expectation stems from the observation that changing individual parameters can cause drastic changes to the patterns and the CNN generally maintains exceptional performance.

Conclusions

In this work, a systematic study of the EBSD operating parameters and their individual effects on the classification performance of a CNN is performed. Despite the CNN being trained from diffraction patterns captured with a fixed geometry and SEM settings, it is found to be resilient over a wide range of conditions. Markedly decreased performance is generally only observed for the most challenging materials to differentiate (e.g., B2 and bcc or L1₂ and fcc). Furthermore, it is encouraging to verify that parameters that effect the time to map an area (e.g., frame averaging or pattern resolution) can be modified to accelerate the process without substantially degrading model performance. For parameters such as tilt, it is reassuring to validate the model performs well over a variety of reasonable parameters. Although the CNN may not achieve high accuracy under all conditions, such as a low number of frames averaged, when used appropriately it remains a highly capable method for assisting the user with phase identification and provides a level of phase differentiation markedly above what state-of-the-art commercial methods are currently capable of. In the current version of the CNN, the parameter settings that cause noteworthy reductions in performance across a majority of space groups are a frame averaging of 1 or utilizing 10 kV accelerating voltage. In the future, training models with patterns collected using a wider variety of operating conditions, particularly those with the largest effect on performance, could result in a model that is even more resilient to some of these types of perturbations to the diffraction patterns; although it may still not be possible to overcome all limitations, such as no frame averaging, owing to significant reductions in the signal-to-noise ratio. A future study should also investigate the co-variation of parameters from the default conditions. Ultimately, we expect the results of this research to encourage the continued development of these tools given the reliability observed and their potential to assist with or automate the analysis of electron diffraction patterns.