Hostname: page-component-cd9895bd7-mkpzs Total loading time: 0 Render date: 2024-12-26T17:16:11.007Z Has data issue: false hasContentIssue false
  • English
  • Français

FROM TEXT TO IMAGES: LINKING SYSTEM REQUIREMENTS TO IMAGES USING JOINT EMBEDDING

Published online by Cambridge University Press:  19 June 2023

Cheng Chen ,
Cody Carroll  and
Beshoy Morkos
Cheng Chen
Affiliation:
University of Georgia
Cody Carroll
Affiliation:
University of Georgia
Beshoy Morkos*
Affiliation:
University of Georgia
*
Morkos, Beshoy, University of Georgia, United States of America, bmorkos@uga.edu

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

Smart manufacturing enterprises rely on adapting to rapid engineering changes while minimizing the generated risk. Making informed decisions related to engineering changes and managing risks against unexpected costs requires more information to be extracted from limited data. However, limited information in early-stage design can come in many forms, namely text and images. The development of innovative design tools and processes to link multisource data together is essential to assist designers in building model-based engineering (MBE) systems. However, the formal computational linking of multisource data is yet to be realized in MBE. We propose a framework to implement transfer learning and integrate domain specific knowledge to bridge this information gap. A synthetic dataset is created using web scraping techniques based on keywords extracted from the requirements. Requirement-image pairs are used to fine tune a contrastive language-image pretraining model to acquire domain knowledge. The results demonstrate how the content of images can be used to indicate all affected requirements for tracing engineering changes in a complex system.

Keywords

RequirementsProduct Lifecycle Management (PLM)ComplexityEngineering ChangeContrastive Language-Image Pre-training (CLIP)
Type
Article
Information
Proceedings of the Design Society , Volume 3: ICED23 , July 2023 , pp. 1985 - 1994
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - ND
This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is unaltered and is properly cited. The written permission of Cambridge University Press must be obtained for commercial re-use or in order to create a derivative work.
Copyright
The Author(s), 2023. Published by Cambridge University Press

References

Castet, C.e.a. (2017), “A point of view from mbse practitioners”, NASA JPL.Google Scholar
Chen, C., Mullis, J. and Morkos, B. (2021), “A topic modeling approach to study design requirements”, in: International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 85383, American Society of Mechanical Engineers, p. V03AT03A021.Google Scholar
Datta, R., Joshi, D., Li, J. and Wang, J.Z. (2008), “Image retrieval: Ideas, influences, and trends of the new age”, ACM Computing Surveys (Csur), Vol. 40 No. 2, pp. 1–60.CrossRefGoogle Scholar
David, M. and Rowe, F. (2016), “What does plms (product lifecycle management systems) manage: Data or documents? complementarity and contingency for smes”, Computers in Industry, Vol. 75, pp. 140–150, http://doi.org/10.1016/jj.compind.2015.05.005.CrossRefGoogle Scholar
Hardoon, D.R., Szedmak, S. and Shawe-Taylor, J. (2004), “Canonical correlation analysis: An overview with application to learning methods”, Neural computation, Vol. 16 No. 12, pp. 2639–2664.CrossRefGoogle ScholarPubMed
Hein, P.H., Kames, E., Chen, C. and Morkos, B. (2021), “Employing machine learning techniques to assess requirement change volatility”, Research in Engineering Design, Vol. 32 No. 2, pp. 245–269.CrossRefGoogle Scholar
Hein, P.H., Kames, E., Chen, C. and Morkos, B. (2022), “Reasoning support for predicting requirement change volatility using complex network metrics”, Journal of Engineering Design, Vol. 33 No. 11, pp. 811–837, http://doi.org/10.1080/09544828.2022.2154051.CrossRefGoogle Scholar
Hein, P.H., Voris, N. and Morkos, B. (2018), “Predicting requirement change propagation through investigation of physical and functional domains”, Research in Engineering Design, Vol. 29 No. 2, pp. 309–328.CrossRefGoogle Scholar
Li, J., Tao, F., Cheng, Y. and Zhao, L. (2015), “Big data in product lifecycle management”, The International Journal of Advanced Manufacturing Technology, Vol. 81 No. 1, pp. 667–684.CrossRefGoogle Scholar
Morkos, B., Joshi, S. and Summers, J.D. (2019), “Investigating the impact of requirements elicitation and evolution on course performance in a pre-capstone design course”, Journal of Engineering Design, Vol. 30 No. 4-5, pp. 155–179, http://doi.org/10.1080/09544828.2019.1605584.CrossRefGoogle Scholar
Morkos, B., Joshi, S., Summers, J.D. and Mocko, G.G. (2010), “Requirements and data content evaluation of industry in-house data management system”, in: International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 44113, pp. 493–503.CrossRefGoogle Scholar
Morkos, B.W. (2012), Computational representation and reasoning supportfor requirements change management in complex system design, Ph.D. thesis, Clemson University.Google Scholar
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. et al. (2021), “Learning transferable visual models from natural language supervision”, in: International Conference on Machine Learning, PMLR, pp. 8748–8763.Google Scholar
Saaksvuori, A. and Immonen, A. (2008), Product lifecycle management systems, Springer.CrossRefGoogle Scholar
Sanh, V., Debut, L., Chaumond, J. and Wolf, T. (2019), “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter”, arXivpreprint arXiv:1910.01108.Google Scholar
Schroff, F., Kalenichenko, D. and Philbin, J. (2015), “Facenet: A unified embedding for face recognition and clustering”, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815–823.CrossRefGoogle Scholar
Shankar, P., Morkos, B. and Summers, J.D. (2012), “Reasons for change propagation: a case study in an automotive oem”, Research in Engineering Design, Vol. 23 No. 4, pp. 291–303, http://doi.org/10.1007/s00163-012-0132-2.CrossRefGoogle Scholar
Sirinam, P., Mathews, N., Rahman, M.S. and Wright, M. (2019), “Triplet fingerprinting: More practical and portable website fingerprinting with n-shot learning”, in: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 1131–1148, http://doi.org/10.1145/3319535.3354217.CrossRefGoogle Scholar
Socher, R., Ganjoo, M., Manning, C.D. and Ng, A. (2013), “Zero-shot learning through cross-modal transfer”, Advances in neural information processing systems, Vol. 26.Google Scholar
Sohn, K. (2016), “Improved deep metric learning with multi-class n-pair loss objective”, Advances in neural information processing systems, Vol. 29.Google Scholar
Stirgwolt, B.W., Mazzuchi, T.A. and Sarkani, S. (2022), “A model-based systems engineering approach for developing modular system architectures”, Journal of Engineering Design, Vol. 33 No. 2, pp. 95–119, http://doi.org/10.1080/09544828.2021.1980203.CrossRefGoogle Scholar
Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H. and Hospedales, T.M. (2018), “Learning to compare: Relation network for few-shot learning”, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1199–1208.CrossRefGoogle Scholar
Tao, F., Zhang, L., Liu, Y., Cheng, Y., Wang, L. and Xu, X. (2015), “Manufacturing service management in cloud manufacturing: overview and future research directions”, Journal of Manufacturing Science and Engineering, Vol. 137 No. 4, http://doi.org/10.1115/L4030510.CrossRefGoogle Scholar
Van Gemert, J. (2003), Retrieving images as text, Ph.D. thesis.Google Scholar
Violante, M.G., Vezzetti, E. and Alemanni, M. (2017), “An integrated approach to support the requirement management (rm) tool customization for a collaborative scenario”, International Journal on Interactive Design andManufacturing (IJIDeM), Vol. 11 No. 2, pp. 191–204, http://doi.org/10.1007/s12008-015-0266-3.CrossRefGoogle Scholar
Wang, L., Liu, Z., Liu, A. and Tao, F. (2021), “Artificial intelligence in product lifecycle management”, The International Journal of Advanced Manufacturing Technology, Vol. 114 No. 3, pp. 771–796.CrossRefGoogle Scholar