Hostname: page-component-cd9895bd7-p9bg8 Total loading time: 0 Render date: 2024-12-26T09:29:16.945Z Has data issue: false hasContentIssue false
  • English
  • Français

A deep learning-based approach to extraction of filler morphology in SEM images with the application of automated quality inspection

Published online by Cambridge University Press:  18 March 2022

Md. Fashiar Rahman Open the ORCID record for Md. Fashiar Rahman [Opens in a new window] ,
Tzu-Liang (Bill) Tseng ,
Jianguo Wu ,
Yuxin Wen  and
Yirong Lin
Md. Fashiar Rahman
Affiliation:
Department of Industrial, Manufacturing and Systems Engineering, The University of Texas at El Paso, TX 79968, USA
Tzu-Liang (Bill) Tseng
Affiliation:
Department of Industrial, Manufacturing and Systems Engineering, The University of Texas, El Paso, TX 79968, USA
Jianguo Wu*
Affiliation:
College of Engineering, Industrial and Systems Engineering, Peking University, Beijing 100871, China
Yuxin Wen
Affiliation:
Department of Electrical and Computer Science, Chapman University, Orange, CA 92866, USA
Yirong Lin
Affiliation:
Department of Mechanical Engineering, The University of Texas, El Paso, TX 79968, USA
*
Author for correspondence: Jianguo Wu, E-mail: j.wu@pku.edu.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

Automatic extraction of filler morphology (size, orientation, and spatial distribution) in Scanning Electron Microscopic (SEM) images is essential in many applications such as automatic quality inspection in composite manufacturing. Extraction of filler morphology greatly depends on accurate segmentation of fillers (fibers and particles), which is a challenging task due to the overlap of fibers and particles and their obscure presence in SEM images. Convolution Neural Networks (CNNs) have been shown to be very effective at object recognition in digital images. This paper proposes an automatic filler detection system in SEM images, utilizing a Mask Region-based CNN architecture. The proposed system can simultaneously classify, detect, and segment fillers in SEM images, making it suitable for morphology analysis of fillers and automatic quality inspection. We also propose a novel SEM image simulation procedure to overcome the data scarcity for training a deep CNN architecture. The proposed filler detection system is trained on the simulated images. It is shown that the trained network can detect and segment fillers with higher accuracy even in the overlapping and obscure situations. The performance and robustness of the proposed system are evaluated using both simulated and real microscopic images.

Keywords

Automated quality inspectionDeep learningDetection and segmentationFiller morphologySEM images
Type
Research Article
Information
AI EDAM , Volume 36 , 2022 , e15
Check for updates
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Chin, RT and Harlow, CA (1982) Automated visual inspection: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 6, 557–573.CrossRefGoogle Scholar
Cirino, M, Friedrich, K and Pipes, R (1988) The effect of fiber orientation on the abrasive wear behavior of polymer composite materials. Wear 121, 127–141.CrossRefGoogle Scholar
Dai, J, Li, Y, He, K and Sun, J (2016) R-fcn: Object detection via region-based fully convolutional networks. In Advances in Neural Information Processing Systems, pp. 379–387.Google Scholar
Doshi, S and Charrier, JM (1989) A simple illustration of structure-properties relationships for short fiber-reinforced thermoplastics. Polymer Composites 10, 28–38.CrossRefGoogle Scholar
Esmaeilian, B, Behdad, S and Wang, B (2016) The evolution and future of manufacturing: a review. Journal of Manufacturing Systems 39, 79–100.CrossRefGoogle Scholar
Felzenszwalb, PF and Huttenlocher, DP (2004) Efficient graph-based image segmentation. International Journal of Computer Vision 59, 167–181.CrossRefGoogle Scholar
Ferguson, M, Ak, R, Lee, Y-TT and Law, KH (2017) Automatic localization of casting defects with convolutional neural networks. In 2017 IEEE International Conference on Big Data (Big Data). IEEE, pp. 1726–1735.CrossRefGoogle Scholar
Ferguson, MK, Ronay, A, Lee, Y-TT and Law, KH (2018) Detection and segmentation of manufacturing defects with convolutional neural networks and transfer learning. Smart and Sustainable Manufacturing Systems 2. doi: 10.1520/SSMS20180033CrossRefGoogle ScholarPubMed
Frangopol, DM and Recek, S (2003) Reliability of fiber-reinforced composite laminate plates. Probabilistic Engineering Mechanics 18, 119–137.CrossRefGoogle Scholar
Ganesh, P, Volle, K, Burks, T and Mehta, S (2019) Deep orange: mask R-CNN based orange detection and segmentation. IFAC-PapersOnLine 52, 70–75.CrossRefGoogle Scholar
Girshick, R (2015) Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448.CrossRefGoogle Scholar
Girshick, R, Donahue, J, Darrell, T and Malik, J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587.CrossRefGoogle Scholar
He, K, Gkioxari, G, Dollár, P and Girshick, R (2017) Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969.CrossRefGoogle Scholar
Huang, L, Geng, L and Peng, H (2015) Microstructurally inhomogeneous composites: is a homogeneous reinforcement distribution optimal? Progress in Materials Science 71, 93–168.CrossRefGoogle Scholar
Huang, J, Rathod, V, Sun, C, Zhu, M, Korattikara, A, Fathi, A, Fischer, I, Wojna, Z, Song, Y, Guadarrama, S and Murphy, K (2017) Speed/accuracy trade-offs for modern convolutional object detectors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7310–7311.CrossRefGoogle Scholar
Illingworth, J and Kittler, J (1988) A survey of the Hough transform. Computer Vision, Graphics, and Image Processing 44, 87–116.CrossRefGoogle Scholar
Image Synthesis With Annotation (2019) Github.Google Scholar
Jung, M-R, Shim, J-H, Ko, B and Nam, J-Y (2008) Automatic cell segmentation and classification using morphological features and Bayesian networks. In Image Processing: Machine Vision Applications, vol. 6813. International Society for Optics and Photonics, p. 68130G.CrossRefGoogle Scholar
Kim, D-H, Sykulski, JK and Lowther, DA (2009) The implications of the use of composite materials in electromagnetic device topology and shape optimization. IEEE Transactions on Magnetics 45, 1154–1157.CrossRefGoogle Scholar
Kothari, S, Chaudry, Q and Wang, MD (2009) Automated cell counting and cluster segmentation using concavity detection and ellipse fitting techniques. In 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. IEEE, pp. 795–798.CrossRefGoogle Scholar
Kovacs, JZ, Andresen, K, Pauls, JR, Garcia, CP, Schossig, M, Schulte, K and Bauhofer, W (2007) Analyzing the quality of carbon nanotube dispersions in polymers using scanning electron microscopy. Carbon 45, 1279–1288.CrossRefGoogle Scholar
Li, X, Tso, SK, Guan, X-P and Huang, Q (2006) Improving automatic detection of defects in castings by applying wavelet technique. IEEE Transactions on Industrial Electronics 53, 1927–1934.CrossRefGoogle Scholar
Lin, T-Y, Maire, M, Belongie, S, Hays, J, Perona, P, Ramanan, D, Dollár, P and Zitnick, CL (2014) Microsoft coco: Common objects in context. In European Conference on Computer Vision. Springer, pp. 740–755.CrossRefGoogle Scholar
Lin, T-Y, Dollár, P, Girshick, R, He, K, Hariharan, B and Belongie, S (2017) Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125.CrossRefGoogle Scholar
Liu, G (1997) A step-by-step method of rule-of-mixture of fiber-and particle-reinforced composite materials. Composite Structures 40, 313–322.CrossRefGoogle Scholar
Liu, Y, Wu, J, Zhou, S and Li, X (2016) Microstructure modeling and ultrasonic wave propagation simulation of A2O6–Al2O3 metal matrix nanocomposites for quality inspection. Journal of Manufacturing Science and Engineering 138, 031008.CrossRefGoogle Scholar
Luo, Y, Liu, L, Huang, Q and Li, X (2017) A novel segmentation approach combining region- and edge-based information for ultrasound images. BioMed Research International 2017, 9157241.CrossRefGoogle ScholarPubMed
Mangan, AP and Whitaker, RT (1999) Partitioning 3D surface meshes using watershed segmentation. IEEE Transactions on Visualization and Computer Graphics 5, 308–321.CrossRefGoogle Scholar
Mangino, E, Carruthers, J and Pitarresi, G (2007) The future use of structural composite materials in the automotive industry. International Journal of Vehicle Design 44, 211–232.CrossRefGoogle Scholar
Mortell, D, Tanner, D and McCarthy, C (2014) In-situ SEM study of transverse cracking and delamination in laminated composite materials. Composites Science and Technology 105, 118–126.CrossRefGoogle Scholar
Mortenson, ME (1999) Mathematics for Computer Graphics Applications. New York, NY: Industrial Press Inc.Google Scholar
Nacereddine, N, Zelmat, M, Belaifa, SS and Tridi, M (2005) Weld defect detection in industrial radiography based digital image processing. Transactions on Engineering Computing and Technology 2, 145–148.Google Scholar
Park, C, Huang, JZ, Huitink, D, Kundu, S, Mallick, BK, Liang, H and Ding, Y (2012 a) A multistage, semi-automated procedure for analyzing the morphology of nanoparticles. IIE Transactions 44, 507–522.CrossRefGoogle Scholar
Park, C, Huang, JZ, Ji, JX and Ding, Y (2012 b) Segmentation, inference and classification of partially overlapping nanoparticles. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1–1.CrossRefGoogle ScholarPubMed
Rahman, MF, Wu, J and Tseng, B (2018) Automated fiber extraction from SEM images with application to quality control of fiber-reinforced composites manufacturing. In International Manufacturing Science and Engineering Conference, vol. 51371. American Society of Mechanical Engineers, p. V003T02A037.CrossRefGoogle Scholar
Rahman, MF, Tseng, T-LB, Pokojovy, M, Qian, W, Totada, B and Xu, H (2021 a) An automatic approach to lung region segmentation in chest X-ray images using adapted U-Net architecture. In Medical Imaging 2021: Physics of Medical Imaging, vol. 11595: International Society for Optics and Photonics, p. 115953I.CrossRefGoogle Scholar
Rahman, MF, Wu, J and Tseng, TLB (2021 b) Automatic morphological extraction of fibers from SEM images for quality control of short fiber-reinforced composites manufacturing. CIRP Journal of Manufacturing Science and Technology 33, 176–187.CrossRefGoogle Scholar
Rosin, PL (1993) Ellipse fitting by accumulating five-point fits. Pattern Recognition Letters 14, 661–669.CrossRefGoogle Scholar
Sarkar, D, Doan, X-T, Ying, Z and Srinivasan, R (2009) In situ particle size estimation for crystallization processes by multivariate image analysis. Chemical Engineering Science 64, 9–19.CrossRefGoogle Scholar
Scholz, MS, Blanchfield, JP, Bloom, LD, Coburn, BH, Elkington, M, Fuller, JD, Gilbert, ME, Muflahi, SA, Pernice, MF, Rae, SI and Trevarthen, JA (2011) The use of composite materials in modern orthopaedic medicine and prosthetic devices: a review. Composites Science and Technology 71, 1791–1803.CrossRefGoogle Scholar
Sharma, P, Diwakar, M and Lal, N (2013) Edge detection using Moore neighborhood. International Journal of Computer Applications 61, 26–30.CrossRefGoogle Scholar
Shi, J and Malik, J (2000) Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 888–905.Google Scholar
Strecker, H (1983) A local feature method for the detection of flaws in automated X-ray inspection of castings. Signal Processing 5, 423–431.CrossRefGoogle Scholar
Tekce, HS, Kumlutas, D and Tavman, IH (2007) Effect of particle shape on thermal conductivity of copper reinforced polymer composites. Journal of Reinforced Plastics and Composites 26, 113–121.CrossRefGoogle Scholar
Toth, C, Koppanyi, Z, Xu, H and Grejner-Brzezinska, D (2017) Localization using region-based convolution neural network: a comparison study. In Proceedings of the 10th International Conference on Mobile Mapping Technology, Cairo, Egypt.Google Scholar
Uribe, S, Belmonte, A, Moreno, F, Llorente, Á, López, JP and Álvarez, F (2019) New access services in HbbTV based on a deep learning approach for media content analysis. AI EDAM 33, 399–415.Google Scholar
Voulodimos, A, Doulamis, N, Doulamis, A and Protopapadakis, E (2018) Deep learning for computer vision: a brief review. Computational Intelligence and Neuroscience 2018, 7068349.CrossRefGoogle ScholarPubMed
Wang, Z (2016) A new approach for segmentation and quantification of cells or nanoparticles. IEEE Transactions on Industrial Informatics 12, 962–971.CrossRefGoogle ScholarPubMed
Wang, G and Liao, TW (2002) Automatic identification of different types of welding defects in radiographic images. NDT & E International 35, 519–528.CrossRefGoogle Scholar
Wessel, J, Heinrich, MP, von Berg, J, Franz, A and Saalbach, A (2019) Sequential rib labeling and segmentation in chest X-ray using Mask R-CNN. arXiv preprint arXiv:1908.08329.Google Scholar
Wu, J, Liu, Y and Zhou, S (2016) Bayesian hierarchical linear modeling of profile data with applications to quality control of nanomanufacturing. IEEE Transactions on Automation Science and Engineering 13, 1355–1366.CrossRefGoogle Scholar
Wu, J, Yuan, Y and Li, X (2017) Size distribution estimation of three-dimensional particle clusters in metal-matrix nanocomposites considering sampling bias. Journal of Manufacturing Science and Engineering 139, 081017.CrossRefGoogle Scholar
Yin, Z, Bise, R, Chen, M and Kanade, T (2010) Cell segmentation in microscopy imagery using a bag of local Bayesian classifiers. In 2010 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. IEEE, pp. 125–128.CrossRefGoogle Scholar
Yu, Z, Brisson, J and Ait-Kadi, A (1994) Prediction of mechanical properties of short kevlar fiber-nylon-6, 6 composites. Polymer Composites 15, 64–73.CrossRefGoogle Scholar
Zhao, K, Kang, J, Jung, J and Sohn, G (2018) Building extraction from satellite images using mask R-CNN with building boundary regularization. In CVPR Workshops, pp. 247–251.CrossRefGoogle Scholar
Zheng, Y and Basart, J (1988) Image analysis, feature extraction and various applied enhancement methods for NDE X-ray images. Review of Progress in QNDE 7, 813–820.Google Scholar