Hostname: page-component-6bf8c574d5-t27h7 Total loading time: 0 Render date: 2025-02-27T14:05:49.074Z Has data issue: false hasContentIssue false
  • English
  • Français

Operationalizing community-based open scientific design research benchmarks: application to model-based architecture design synthesis

Published online by Cambridge University Press:  16 May 2024

Romain Pinquié ,
Lionel Roucoules ,
Pierre-Alain Yvars  and
Raphaël Chenouard
Romain Pinquié*
Affiliation:
Univ. Grenoble Alpes, CNRS, Grenoble INP, G-SCOP, France
Lionel Roucoules
Affiliation:
Arts et Métiers ParisTech, France
Pierre-Alain Yvars
Affiliation:
ISAE-Supméca, France
Raphaël Chenouard
Affiliation:
École Centrale de Nantes, France
*
romain.pinquie@grenoble-inp.fr

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

The point has repeatedly been made that validation is a crucial success factor in demonstrating the scientific contribution and ensuring the adoption of results. Still, researchers in design science validate their research findings too infrequently. We must all evaluate our claimed contributions on open benchmarks to improve validation quality and foster cumulative research. In this paper, we propose a meta-model to standardise and operationalise the concept of open scientific benchmarks in design science and to guide communities of researchers in the co-development of scientific benchmarks.

Keywords

open sciencedesign researchbenchmarkingmodel-based systems engineering (MBSE)product architecture
Type
Design Theory and Research Methods
Information
Proceedings of the Design Society , Volume 4: DESIGN 2024 , May 2024 , pp. 95 - 104
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - ND
This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is unaltered and is properly cited. The written permission of Cambridge University Press must be obtained for commercial re-use or in order to create a derivative work.
Copyright
The Author(s), 2024.

References

Albert, A., Korkiat, S., Christian, S. and Wolfgang, B. (2009), “A Method to define a Product Architecture Early in Product Development Using the Contact and Channel Model”, Stanford University, Stanford, CA, USA, pp. 241–252.Google Scholar
Barth, A., Caillaud, E. and Rose, B. (2011), “How to validate research in engineering design?”, 18th International Conference on Engineering Design (ICED 11), Impacting Society through Engineering Design, , Vol. 2: Design Theory and Research Methodology, Copenhagen, Denmark, pp. 41–50.Google Scholar
Blessing, L.T.M. and Chakrabarti, A. (2009), DRM, a Design Research Methodology, Springer London, London, https://dx.doi.org/10.1007/978-1-84882-587-1.CrossRefGoogle Scholar
Boniol, F., Wiels, V., Aït-Ameur, Y. and Schewe, K.-D. (2017), “The landing gear case study: challenges and experiments”, International Journal on Software Tools for Technology Transfer, Vol. 19 No. 2, pp. 133–140, https://dx.doi.org/10.1007/s10009-016-0431-4.CrossRefGoogle Scholar
Broodney, H., Dotan, D., Greenberg, L. and Masin, M. (2012), “Generic Approach for Systems Design Optimization in MBSE”, INCOSE International Symposium, Vol. 22 No. 1, pp. 184–200, https://dx.doi.org/10.1002/j.2334-5837.2012.tb01330.x.CrossRefGoogle Scholar
Cagan, J., Campbell, M.I., Finger, S. and Tomiyama, T. (2005), “A Framework for Computational Design Synthesis: Model and Applications”, Journal of Computing and Information Science in Engineering, Vol. 5 No. 3, pp. 171–181, https://dx.doi.org/10.1115/1.2013289.CrossRefGoogle Scholar
Cantamessa, M. (2003), “An empirical perspective upon design research”, Journal of Engineering Design, Vol. 14 No. 1, pp. 1–15, https://dx.doi.org/10.1080/0954482031000078126.CrossRefGoogle Scholar
Seepersad, Carolyn C., Pedersen, K., Emblemsvåg, J., Bailey, R. and Allen, J.K. (2006), “The Validation Square: How Does One Verify and Validate a Design Method?”, Decision Making in Engineering Design, ASME Press, pp. 303–313, https://dx.doi.org/10.1115/1.802469.ch25.CrossRefGoogle Scholar
Cash, P.J. (2018), “Developing theory-driven design research”, Design Studies, Vol. 56, pp. 84–119, https://dx.doi.org/10.1016/j.destud.2018.03.002.CrossRefGoogle Scholar
Eckert, C., Clarkson, P.J., and Stacey. (2003), “The spiral of applied research: A methodological view on integrated design research”, Proceedings of the 14th International Conference on Engineering Design (ICED’03), Stockholm, Sweden.Google Scholar
Eckert, C., Clarkson, P.J. and Stacey, M. (2004), “The lure of the measurable in design research”, presented at the International Design Conference - DESIGN, Dubrovnik, Croatia.Google Scholar
Eisenmann, M., Grauberger, P., Üreten, S., Krause, D. and Matthiesen, S. (2021), “Design method validation–an investigation of the current practice in design research”, Journal of Engineering Design, Taylor and Francis Ltd., Vol. 32 No. 11, pp. 621–645, https://dx.doi.org/10.1080/09544828.2021.1950655.Google Scholar
Escudero-Mancebo, D., Fernández-Villalobos, N., Martín-Llorente, Ó. and Martínez-Monés, A. (2023), “Research methods in engineering design: a synthesis of recent studies using a systematic literature review”, Research in Engineering Design, Vol. 34 No. 2, pp. 221–256, https://dx.doi.org/10.1007/s00163-022-00406-y.CrossRefGoogle Scholar
Formentini, G., Bouissiere, F., Cuiller, C., Eric, P. and Claudio, D. (2022), “CDFA method : a way to assess assembly and installation performance of aircraft system architectures at the conceptual design”, Research in Engineering Design, Springer London, Vol. 33 No. 1, pp. 31–52, https://dx.doi.org/10.1007/s00163-021-00378-5.CrossRefGoogle Scholar
Frey, D.D. and Dym, C.L. (2006), “Validation of design methods: lessons from medicine”, Research in Engineering Design, Vol. 17 No. 1, pp. 45–57, https://dx.doi.org/10.1007/s00163-006-0016-4.CrossRefGoogle Scholar
Gericke, K., Eckert, C., Campean, F., Clarkson, P.J., Flening, E., Isaksson, O., Kipouros, T., et al. (2020), “Supporting designers: moving from method menagerie to method ecosystem”, Design Science, Vol. 6, p. e21, https://dx.doi.org/10.1017/dsj.2020.21.CrossRefGoogle Scholar
Gericke, K., Eckert, C. and Stacey, M. (2022), “Elements of a design method – a basis for describing and evaluating design methods”, Design Science, Cambridge University Press, Vol. 8, p. e29, https://dx.doi.org/10.1017/dsj.2022.23.Google Scholar
Gericke, K., Meissner, M. and Paetzold, K. (2003), “Understanding the context of product development”, Proceedings of the 19th International Conference on Engineering Design (ICED13) Design For Harmonies, Vol. 3: Design Organisation and Management., Seoul, Korea.Google Scholar
Hartmann, C., Chenouard, R., Mermoz, E. and Bernard, A. (2018), “A framework for automatic architectural synthesis in conceptual design phase”, Journal of Engineering Design, Vol. 29 No. 11, pp. 665–689, https://dx.doi.org/10.1080/09544828.2018.1532494.CrossRefGoogle Scholar
Hein, A.M. and Lamé, G. (2020), “Evaluating engineering design methods: taking inspiration from software engineering and the health sciences”, Proceedings of the Design Society: DESIGN Conference, Vol. 1, pp. 1901–1910, https://dx.doi.org/10.1017/dsd.2020.317.CrossRefGoogle Scholar
Hirsch, T. and Hofer, B. (2022), “A systematic literature review on benchmarks for evaluating debugging approaches”, Journal of Systems and Software, Vol. 192, p. 111423, https://dx.doi.org/10.1016/j.jss.2022.111423.CrossRefGoogle Scholar
Isaksson, O., Eckert, C., Panarotto, M. and Malmqvist, J. (2020), “You need to focus to validate”, Proceedings of the Design Society: DESIGN Conference, Vol. 1, pp. 31–40, https://dx.doi.org/10.1017/dsd.2020.116.CrossRefGoogle Scholar
Kryssanov, V.V., Tamaki, H. and Ueda, K. (1999), “Synthesis and emergence in engineering design problem solving”, Proceedings First International Symposium on Environmentally Conscious Design and Inverse Manufacturing, presented at the Proceedings First International Symposium on Environmentally Conscious Design and Inverse Manufacturing, IEEE, Tokyo, Japan, pp. 690–695, https://dx.doi.org/10.1109/ECODIM.1999.747699.CrossRefGoogle Scholar
Lempia, D. and Miller, S. (2009), “Requirements Engineering Management Handbook, U.S. Department of Transportation, Federal Aviation Administration”.Google Scholar
Paehler, Lukas, Sankowski, Olga, Ureten, Selin, Eisenmann, Matthias, and Krause, Dieter. (2023), “Impact of Method Users on the Application of Design Methods - Assessing the Role of Method-related Background Knowledge”, DS 125: Proceedings of the 34th Symposium Design for X (DFX2023), presented at the The 34th Symposium Design for X, The Design Society, pp. 055–064, https://dx.doi.org/10.35199/dfx2023.06.CrossRefGoogle Scholar
Menu, J. and Nicolai, M. (2017), “Ast 2017 a Framework for Automated Design, Verification , and Simulation of Electrical Power Systems for Aircraft”, International Workshop on Aircraft System Technologies, pp. 1–10.Google Scholar
Menu, J., Nicolai, M. and Zeller, M. (2018), “Designing Fail-Safe Architectures for Aircraft Electrical Power Systems”, 2018 AIAA/IEEE Electric Aircraft Technologies Symposium, EATS 2018, pp. 1–14, https://dx.doi.org/10.2514/6.2018-5032.CrossRefGoogle Scholar
Nuzzo, P., Xu, H., Ozay, N., Finn, J.B., Sangiovanni-Vincentelli, A.L., Murray, R.M., Donzé, A., et al. (2014), “A contract-based methodology for aircraft electric power system design”, IEEE Access, Vol. 2, pp. 1–25, https://dx.doi.org/10.1109/ACCESS.2013.2295764.CrossRefGoogle Scholar
Pailhès, J., Sallaou, M., Nadeau, J.-P. and Fadel, G.M. (2011), “Energy Based Functional Decomposition in Preliminary Design”, Journal of Mechanical Design, Vol. 133 No. 5, p. 051011, https://dx.doi.org/10.1115/1.4004193.CrossRefGoogle Scholar
Pinquié, R., Duigou, J.L., Grimal, L. and Roucoules, L. (2022), “An open science platform for benchmarking engineering design researches”, Procedia CIRP, Vol. 109, pp. 472–477, https://dx.doi.org/10.1016/j.procir.2022.05.280.CrossRefGoogle Scholar
Santos, Ribeiro Dos, Hany Saleh, C.A., Schrijvers, A., and Nicolai, T., M. (2019), “CONDEnSe: Contract Based Design Synthesis”, Proceedings - 2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems, MODELS 2019, pp. 250–260, https://dx.doi.org/10.1109/MODELS.2019.00009.CrossRefGoogle Scholar
Rosich, A., Berx, K. and Pinte, G. (2016), “Model-based design synthesis: application to optimal air-compressor system design”, IFAC-PapersOnLine, Vol. 49 No. 21, pp. 416–422, https://dx.doi.org/10.1016/j.ifacol.2016.10.604.CrossRefGoogle Scholar
Sim, S.E., Easterbrook, S. and Holt, R.C. (2003), “Using benchmarking to advance research: a challenge to software engineering”, 25th International Conference on Software Engineering, 2003. Proceedings., presented at the 25th International Conference on Software Engineering, 2003. Proceedings., IEEE, Portland, OR, USA, pp. 74–83, https://dx.doi.org/10.1109/ICSE.2003.1201189.CrossRefGoogle Scholar
Suh, N.P. (2001), Axiomatic Design : Advances and Applications, Oxford University Press.Google Scholar
Thiyagalingam, J., Shankar, M., Fox, G. and Hey, T. (2022), “Scientific machine learning benchmarks”, Nature Reviews Physics, Vol. 4 No. 6, pp. 413–420, https://dx.doi.org/10.1038/s42254-022-00441-7.CrossRefGoogle Scholar
Tomiyama, T. (2020), “How to write a good STC Dn paper”, CIRP Annals Manufacturing Technology.Google Scholar
Vermaas, P.E. (2014), “Design Theories, Models and Their Testing: On the Scientific Status of Design Research”, An Anthology of Theories and Models of Design, Springer London, London, pp. 47–66, https://dx.doi.org/10.1007/978-1-4471-6338-1_2.CrossRefGoogle Scholar
Yvars, P.A., Lafon, P. and Zimmer, L. (2009), “Optimization of mechanical system: Contribution of constraint satisfaction method”, 2009 International Conference on Computers & Industrial Engineering, presented at the Industrial Engineering (CIE39), IEEE, Troyes, France, pp. 1379–1384, https://dx.doi.org/10.1109/ICCIE.2009.5223678.CrossRefGoogle Scholar
Yvars, P.-A. and Zimmer, L. (2021), “A Model-based Synthesis approach to system design correct by construction under environmental impact requirements”, Procedia CIRP, Vol. 103, pp. 85–90, https://dx.doi.org/10.1016/j.procir.2021.10.013.CrossRefGoogle Scholar
Yvars, P.-A. and Zimmer, L. (2022), “Towards a correct by construction design of complex systems: The MBSS approach”, Procedia CIRP, Vol. 109, pp. 269–274, https://dx.doi.org/10.1016/j.procir.2022.05.248.CrossRefGoogle Scholar