Hostname: page-component-cd9895bd7-dk4vv Total loading time: 0 Render date: 2024-12-28T01:19:38.395Z Has data issue: false hasContentIssue false
  • English
  • Français

Towards a process for the creation of synthetic training data for AI-computer vision models utilizing engineering data

Published online by Cambridge University Press:  16 May 2024

Sebastian Schwoch ,
Maximilian Peter Dammann ,
Johannes Georg Bartl ,
Maximilian Kretzschmar ,
Bernhard Saske  and
Kristin Paetzold-Byhain
Sebastian Schwoch*
Affiliation:
Technische Universität Dresden, Germany
Maximilian Peter Dammann
Affiliation:
Technische Universität Dresden, Germany
Johannes Georg Bartl
Affiliation:
Technische Universität Dresden, Germany
Maximilian Kretzschmar
Affiliation:
Technische Universität Dresden, Germany
Bernhard Saske
Affiliation:
Technische Universität Dresden, Germany
Kristin Paetzold-Byhain
Affiliation:
Technische Universität Dresden, Germany
*
sebastian.schwoch@tu-dresden.de

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

Artificial Intelligence-based Computer Vision models (AI-CV models) for object detection can support various applications over the entire lifecycle of machines and plants such as monitoring or maintenance tasks. Despite ongoing research on using engineering data to synthesize training data for AI-CV model development, there is a lack of process guidelines for the creation of such data. This paper proposes a synthetic training data creation process tailored to the particularities of an engineering context addressing challenges such as the domain gap and methods like domain randomization.

Keywords

artificial intelligence (AI)synthetic training dataannotation automationdomain gapprocess improvement
Type
Artificial Intelligence and Data-Driven Design
Information
Proceedings of the Design Society , Volume 4: DESIGN 2024 , May 2024 , pp. 2237 - 2246
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - ND
This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is unaltered and is properly cited. The written permission of Cambridge University Press must be obtained for commercial re-use or in order to create a derivative work.
Copyright
The Author(s), 2024.

References

Ahmad, K., Abdelrazek, M., Arora, C., Bano, M. and Grundy, J. (2023), “Requirements engineering for artificial intelligence systems: A systematic mapping study”, Information and Software Technology, Vol. 158, p. 107176. https://dx.doi.org/10.1016/j.infsof.2023.107176CrossRefGoogle Scholar
Assadzadeh, A., Arashpour, M., Brilakis, I., Ngo, T. and Konstantinou, E. (2022), “Vision-based excavator pose estimation using synthetically generated datasets with domain randomization”, Automation in Construction, Vol. 134, p. 104089. https://dx.doi.org/10.1016/j.autcon.2021.104089CrossRefGoogle Scholar
Chen, Y., Li, W., Sakaridis, C., Dai, D. and van Gool, L. (2018), Domain Adaptive Faster R-CNN for Object Detection in the Wild. https://dx.doi.org/10.48550/arXiv.1803.03243CrossRefGoogle Scholar
Dammann, M.P., Steger, W. and Stelzer, R. (2022), “Automated and Adaptive Geometry Preparation for AR/VR-Applications”, Journal of Computing and Information Science in Engineering, Vol. 22 No. 3. https://dx.doi.org/10.1115/1.4053327CrossRefGoogle Scholar
Denninger, M., Sundermeyer, M., Winkelbauer, D., Zidan, Y. and Olefir, D., et al. (2019), BlenderProc. https://dx.doi.org/10.48550/arXiv.1911.01911CrossRefGoogle Scholar
DIN Deutsches Institut für Normung e.V. (2020), DIN SPEC 13266:2020-04, Leitfaden für die Entwicklung von Deep-Learning-Bilderkennungssystemen, Beuth Verlag GmbH, Berlin. https://dx.doi.org/10.31030/3134557Google Scholar
Downs, L., Francis, A., Koenig, N., Kinman, B. and Hickman, R., et al. (2022), Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items. https://dx.doi.org/10.48550/arXiv.2204.11918CrossRefGoogle Scholar
Drost, B., Ulrich, M., Bergmann, P., Hartinger, P. and Steger, C. (2017), “Introducing MVTec ITODD — A Dataset for 3D Object Recognition in Industry”, paper presented at 2017 IEEE International Conference on Computer Vision Workshop (ICCVW), 22.10.-29.10.2017, Venice, Italy. https://dx.doi.org/10.1109/ICCVW.2017.257CrossRefGoogle Scholar
Everingham, M., van Gool, L., Williams, C.K.I., Winn, J. and Zisserman, A. (2010), “The Pascal Visual Object Classes (VOC) Challenge”, International Journal of Computer Vision, Vol. 88 No. 2, pp. 303–338. https://dx.doi.org/10.1007/s11263-009-0275-4CrossRefGoogle Scholar
He, J., Yang, S., Yang, S., Kortylewski, A. and Yuan, X., et al. (2022), “PartImageNet: A Large, High-Quality Dataset of Parts”, in Avidan, S., Brostow, G., Cissé, M., Farinella, G.M. and Hassner, T. (Eds.), Computer Vision – ECCV 2022, Lecture notes in computer science, Vol. 13668, Springer Nature Switzerland, Cham, pp. 128–145. https://dx.doi.org/10.48550/arXiv.2112.00933CrossRefGoogle Scholar
He, K., Gkioxari, G., Dollár, P. and Girshick, R. (2017), Mask R-CNN. https://dx.doi.org/10.48550/arXiv.1703.06870CrossRefGoogle Scholar
Hesenius, M., Schwenzfeier, N., Meyer, O., Koop, W. and Gruhn, V. (2019), “Towards a Software Engineering Process for Developing Data-Driven Applications”, in 2019 IEEE/ACM 7th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering, 5/28/-5/28/2019, Montreal, QC, Canada, IEEE, Piscataway, NJ, pp. 35–41. https://dx.doi.org/10.1109/RAISE.2019.00014CrossRefGoogle Scholar
Horváth, D., Erdős, G., Istenes, Z., Horváth, T. and Földi, S. (2023), “Object Detection Using Sim2Real Domain Randomization for Robotic Applications”, IEEE Transactions on Robotics, Vol. 39 No. 2, pp. 1225–1243. https://dx.doi.org/10.1109/tro.2022.3207619CrossRefGoogle Scholar
Hussain, M. (2023), “YOLO-v1 to YOLO-v8, the Rise of YOLO and Its Complementary Nature toward Digital Manufacturing and Industrial Defect Detection”, Machines, Vol. 11 No. 7, p. 677. https://dx.doi.org/10.3390/machines11070677CrossRefGoogle Scholar
Jain, A., Patel, H., Nagalapatti, L., Gupta, N. and Mehta, S., et al. (2020), “Overview and Importance of Data Quality for Machine Learning Tasks”, in Gupta, R., Liu, Y., Shah, M., Rajan, S., Tang, J. and Prakash, B.A. (Eds.), Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020/07/06-10, USA, ACM, New York, USA, pp. 3561–3562. https://dx.doi.org/10.1145/3394486.3406477Google Scholar
Jocher, G. and Waxmann, S. (2023), “Ultralytics YOLOv8 Docs. Datasets Overview”, available at: https://docs.ultralytics.com/datasets/ (accessed 13 February 2024)Google Scholar
Kohtala, S. and Steinert, M. (2021), “Leveraging synthetic data from CAD models for training object detection models – a VR industry application case”, Procedia CIRP, Vol. 100, pp. 714–719. https://dx.doi.org/10.1016/j.procir.2021.05.092CrossRefGoogle Scholar
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L. and Girshick, R., et al. (2014), Microsoft COCO: Common Objects in Context. https://dx.doi.org/10.48550/arXiv.1405.0312CrossRefGoogle Scholar
Mayer, N., Ilg, E., Fischer, P., Hazirbas, C. and Cremers, D., et al. (2018), “What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?”, International Journal of Computer Vision, Vol. 126 No. 9, pp. 942–960. https://dx.doi.org/10.1007/s11263-018-1082-6Google Scholar
Microsoft (2023), “What is the Team Data Science Process?”, available at: https://learn.microsoft.com/en-us/azure/architecture/data-science-process/overview (accessed 13 February 2024)Google Scholar
Ono, T., Suzuki, A. and Tamukoh, H. (2023), “An Effective Method for Minimizing Domain Gap in Sim2Real Object Recognition Using Domain Randomization”, Proceedings of International Conference on Artificial Life and Robotics, Vol. 28, pp. 420–424. https://dx.doi.org/10.5954/icarob.2023.os17-6CrossRefGoogle Scholar
Orhei, C., Vert, S., Mocofan, M. and Vasiu, R. (2021), “End-To-End Computer Vision Framework: An Open-Source Platform for Research and Education”, Sensors (Basel, Switzerland), Vol. 21 No. 11. https://dx.doi.org/10.3390/s21113691CrossRefGoogle ScholarPubMed
Park, K., Lee, H., Yang, H. and Oh, S.-Y. (2020), “Improving Instance Segmentation using Synthetic Data with Artificial Distractors”, in ICCAS 2020: 2020 20th International Conference on Control, Automation and Systems proceedings October 13 (Tue)-16 (Fri), 2020, BEXCO, Busan, Korea, 10/13/2020 - 10/16/2020, Busan, Korea (South), IEEE, Piscataway, NJ, pp. 22–26. https://dx.doi.org/10.23919/ICCAS50221.2020.9268390CrossRefGoogle Scholar
Pasanisi, D., Rota, E., Ermidoro, M. and Fasanotti, L. (2023), “On Domain Randomization for Object Detection in real industrial scenarios using Synthetic Images”, Procedia Computer Science, Vol. 217, pp. 816–825. https://dx.doi.org/10.1016/j.procs.2022.12.278CrossRefGoogle Scholar
Salas, A.J.C., Meza-Lovon, G., Fernandez, M.E.L. and Raposo, A. (2020), “Training with synthetic images for object detection and segmentation in real machinery images”, in 2020 33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), 07.11.2020 - 10.11.2020, Recife/Porto de Galinhas, Brazil, IEEE, pp. 226–233. https://dx.doi.org/10.1109/SIBGRAPI51738.2020.00038CrossRefGoogle Scholar
Schulz, M., Neuhaus, U., Kaufmann, J., Kühnel, S. and Alekozai, E.M., et al. (2022), DASC-PM v1.1: A process model for data science projects, Nordakademie gAG Hochschule der Wirtschaft; Universitäts- und Landesbibliothek Sachsen-Anhalt, Elmshorn, Halle (Saale)Google Scholar
Schwoch, S., Leidich, J., Layer, M., Saske, B. and Paetzold-Byhain, K., et al. (2023), “A conceptual framework for information linkage and exchange throughout the lifecycle of process plants”, in DS 125: Proceedings of the 34th Symposium Design for X (DFX2023), 14th and 15th September 2023, The Design Society, pp. 245–256. https://dx.doi.org/10.35199/dfx2023.25CrossRefGoogle Scholar
Stelzer, R., Steger, W. and Petermann, D. (2012), “The VR Session Manager: A Tool to Co-Ordinate a Collaborative Product Development Process in a Virtual Environment”, in Proceedings of the ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference - 2012, 8/12/2012 - 8/15/2012, Chicago, Illinois, USA, ASME, New York, NY, pp. 1517–1525. https://dx.doi.org/10.1115/DETC2012-70998CrossRefGoogle Scholar
Gould, Stephen (2012), “DARWIN: A Framework for Machine Learning and Computer Vision Research and Development”, Journal of Machine Learning Research, Vol. 13 No. 113, pp. 3533–3537Google Scholar
Tremblay, J., Prakash, A., Acuna, D., Brophy, M. and Jampani, V., et al. (2018), Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization. https://dx.doi.org/10.48550/arXiv.1804.06516CrossRefGoogle Scholar
Valtchev, S.Z. and Wu, J. (2021), “Domain randomization for neural network classification”, Journal of big data, Vol. 8 No. 1, p. 94. https://dx.doi.org/10.1186/s40537-021-00455-5CrossRefGoogle ScholarPubMed
Wong, M.Z., Kunii, K., Baylis, M., Ong, W.H., Kroupa, P. and Koller, S. (2019), “Synthetic dataset generation for object-to-model deep learning in industrial applications”, PeerJ. Computer science, Vol. 5, e222. https://dx.doi.org/10.7717/peerj-cs.222CrossRefGoogle ScholarPubMed
Zhou, L., Zhang, L. and Konz, N. (2023), “Computer Vision Techniques in Manufacturing”, IEEE Transactions on Systems, Man, and Cybernetics: Systems, Vol. 53 No. 1, pp. 105–117. https://dx.doi.org/10.1109/TSMC.2022.3166397CrossRefGoogle Scholar