Defining success in open source hardware development projects: a survey of practitioners

Abstract

Recent years have seen the rise of citizens as contributors to hardware product creation. This trend has increased attention to open source hardware (OSH): a phenomenon that extends the intellectual property management and development practices in open source software (OSS) into the design of physical objects. OSH projects are different from OSS projects due to product type, and distinct from traditional closed source new product development (NPD) ones due to their openness. These differences challenge the degree of applicability of existing project success definitions in the OSH context. To investigate project success in OSH, we conducted a qualitative survey with practitioners. We report characteristics of successful OSH projects through three identified themes: (a) value creation – the big-picture impact, (b) quality of output – the quality of the hardware and accompanying documentation and (c) project process – activities that contribute to success. We contextualise by comparing OSH with selected literature on the success of OSS and NPD project management. While our study confirms a similarity between OSS and OSH in defining project success, it also highlights themes that are uniquely important to the latter. These findings are helpful for OSH development practice and could provide lessons for OSS development and closed source NPD.

1. Introduction

In recent years, we have observed a proliferation of open source hardware (OSH) initiatives, with some developing profitable businesses. At the time of writing, ¹ the Open Source Hardware Association has certified 1663 OSH projects ² and the Open Know-How search engine lists 486 OSH projects. ³ A 2018 study analysed over 200 OSH projects (Bonvoisin et al. 2018), while OSH business models have also been discussed in the literature (Pearce 2017; Li & Seering 2019). Pearce (2016) states that open source scientific hardware can achieve between 100% and 1000% return on investment after just a few months.

Success, in its traditional definition in common language, relates to the accomplishment of goals. Success criteria are important in any project, as they give its participants a focus for their efforts (Yu, Flett & Bowers 2005). Success criteria can help OSH communities ‘build effective forms of collective action and self-organisation’ and ‘effectively create and capture value’ (Troxler 2013). They can also aid in the formation of ‘a consistent identity and a set of commonly accepted best practices’ to help the OSH phenomenon become more mature (Bonvoisin et al. 2020). This is because employing best practices can help steer a project towards success (Griffin 1997).

Despite its relevance, there is a lack of comprehensive understanding of how success is defined in OSH projects, which has the potential to benefit both research and practice. A few publications attempt to suggest good practices or measures of impact (e.g., van der Bij et al. 2013; GOSH 2016; Bonvoisin & Mies 2018) but those only provide a partial view of success. This article addresses this shortcoming by investigating how practitioners characterise success in OSH projects (Objective 1) and identifying similarities and differences with other modes of product development (Objective 2).

Firstly, we explore what characterises successful OSH projects, drawing insights from a qualitative survey of 30 OSH practitioners.

Objective 1: Understand success in OSH projects

To fulfil the first objective, we must answer the following research questions (RQs):

RQ1. What characteristics and practices are present in successful OSH projects?

RQ2. What metrics can be used to measure success in OSH projects?

To answer these questions, we collected and analysed data on the opinions of practitioners, who reflected on their experience with OSH projects. We define ‘practitioner’ as someone who has experience participating in an OSH project, has a real intention to do so or has research experience in the subject.

Secondly, to identify the distinctiveness of OSH project success, we compare our findings to characterisations of success found in selected open source software (OSS) and closed source new product development (NPD) project management literature.

Objective 2: Identify aspects of success that are uniquely important to OSH projects

To fulfil objective 2, we asked the following RQs:

RQ3. Does success look different in OSH projects than in OSS?

RQ4. How does success in OSH projects compare to success in NPD projectmanagement?

We answered these RQs by comparing our findings with selected literature on OSS and NPD project management.

This article proceeds as follows: Section 1.1 reviews the selected relevant literature and describes the research gap addressed by the research objectives and questions. Section 1.2 summarises the significance of this research. Section 2 outlines the methodological approach for fulfilling the two objectives of the study. Section 3 presents the characteristics of successful OSH projects according to the opinions of the OSH practitioners surveyed (Objective 1). Section 4 discusses the findings, compares them with OSS and NPD success (Objective 2), and presents the study limitations and avenues for future work. Finally, Section 5 concludes by summarising and highlighting the practical implications of this study.

1.1. Background and literature review

This section is devoted to laying the basis of our discussion and analysis. It starts by defining relevant concepts, including ‘open source’ (section What is an open source product?), and ‘project openness’ (section Product versus process openness), and how they apply to OSH projects. We then identify the gap in the literature by outlining extant research on success in OSH, OSS and closed source NPD (section Literature gap).

What is an open source product ⁴?

When a product is open source, it means that its users have four freedoms: (a) to use it for any purpose, (b) to study it, (c) to make and redistribute copies of it and (d) to make changes to it and share them (Stallman 2002). The articulation of these fundamental freedoms originated in the early days of software development when developers openly shared source code and built on each other’s work (Stallman 2002). Software that respects these freedoms through open source licencing is referred to as OSS. There are many examples of OSS, including the Mozilla Firefox web browser, ⁵ the WordPress content management system, ⁶ and the Linux kernel ⁷ on which many enterprises and mobile operating systems are based.

These freedoms are also reflected in OSH. Specifically, the definition of OSH by the Open Source Hardware Association (2018) states that “[OSH] is hardware whose design is made publicly available so that anyone can study, modify, distribute, make, and sell the design or hardware based on that design”.

While access to source code is needed to practice those freedoms for software, what constitutes the ‘source’ of OSH is less well-defined (Bonvoisin et al. 2017). More recently, the OSH specification DIN SPEC 3105 (n.d.) describes the requirements for what constitutes an adequate ‘source’ in OSH. It also transposes the four freedoms of open source into the four ‘rights’ of OSH: the right to study, to modify, to make and to distribute (Bonvoisin et al. 2020), in line with the OSH Definition (Open Source Hardware Association 2018). For this article, we consider the source to be all necessary documentation – such as blueprints, computer-aided design (CAD) files, or bills of materials (BoMs) – which enable a person to exercise the four rights of OSH.

Prominent examples of OSH include the RepRap 3D printer, ⁸ the AudioMoth environmental sensor, ⁹ the Opentrons lab automation system ¹⁰ and the FOSSASAT ¹¹ series of satellites first launched into space in December 2019. ¹² The achievements of OSH projects have garnered academic interest, as reflected by the emergence of peer-reviewed journals dedicated to OSH such as the Journal of Open Hardware and HardwareX. The development of OSH is a unique type of product development that enables the incorporation of users in the design process. Thus, it is a highly relevant topic in design science (Papalambros 2015).

Product versus process openness

The OSH phenomenon is co-occurring with a “paradigm shift in industrial value creation”, which is often observed through novel processes that are outside the umbrella of traditional economics (Moritz, Redlich & Wulfsberg 2018). These processes, which include “networking, knowledge sharing, collaboration, co-creation and decentralisation” (Moritz et al. 2018), are part of the ‘bottom-up economics’ concept (Wulfsberg, Redlich & Bruhns 2011).

The emergence of OSH sets the scene for new, ‘open’ product development practices: participative, democratic, community-based, and open to the participation of any interested person, regardless of background. OSH development projects (hereinafter referred to as OSH projects) can be characterised by their degree of openness, which has three factors (Balka, Raasch & Herstatt 2014):

1. (i) Transparency: any person can have unrestricted access to product information.

2. (ii) Accessibility: any person can take part in the product development process.

3. (iii) Replicability: any person can physically reproduce the product if following the design guidelines.

Additionally, Huizingh (2011) identified two types of ‘openness’: product openness and process openness. These relate to transparency, accessibility and replicability and indicate that they are not binary states, but rather lie on a spectrum. In other words, OSH projects have a certain level of transparency, accessibility and replicability.

Product openness refers to how much of the design documentation (CAD files, BoMs, etc.) of the final product are open source as defined in section What is an open source product?. The two extrema of the spectrum of product openness are closed source hardware and OSH. The former are physical products for which no documentation is publicly available, and people are not allowed to make and distribute copies or make changes to the designs. The latter are products for which all design documentation is available with open source licencing (Bonvoisin et al. 2018), therefore granting the public the four freedoms of open source (section What is an open source product?). Product openness relates to transparency and replicability as defined by Balka et al. (2014).

Process openness relates to the ‘intention’ of assembling a group of voluntary participants to take part in the design process. To have process openness in a project, there must be product development processes that allow interested persons to participate (Bonvoisin et al. 2018). Design projects lie within a spectrum of process openness, with the extrema being completely closed design and completely open design. The latter involves product development which is open to participation by any external person, while the former allows no external participation at all. Process openness relates to accessibility according to the definition of Balka et al. (2014).

The Open Source Hardware Association (2018) definition and DIN SPEC 3105 have requirements for product openness only, with process openness left as an optional best practice. However, Bonvoisin & Mies (2018) proposed a tool called Open-o-Meter, which does uses process and product openness criteria, for assessing the extent of openness of an OSH project. The relevance of process openness for project success should be further explored.

Literature gap

Research regarding the development of OSH is still in its infancy. The limited number of published studies that exist have focused on describing this field and highlighting emerging issues. Boisseau, Bouchard & Omhover (2017) propose a design process model using a grounded theory approach; Bonvoisin et al. (2018) investigate participation in OSH projects; Dai et al. (2020) highlight issues in knowledge management of OSH communities; and Balka, Raasch & Herstatt (2009) compare OSH development to OSS development and present project characteristics.

However, when it comes to OSH project success, there is currently very little literature. Some effort has been made to standardise technical documentation for OSH projects, that is, DIN SPEC 3105. This could be related to success, but is only limited to technical documentation, not other project practices. Moritz et al. (2018), though aiming to identify best practices in OSH projects, effectively provide merely a description of OSH projects and companies (e.g., licencing selection, community size and community roles). Bonvoisin & Mies (2018) present the Open-o-Meter, a tool for measuring the ‘openness’ of an OSH project, which offers only a partial view of what might constitute success. The Open Impact Toolkit provides a set of metrics for measuring the impact for OSH projects (GOSH 2016). It gives some interesting examples of what factors (in the form of metrics) could affect ‘project impact’, such as usage of the hardware and derivative works. However, its definition of ‘project impact’ is vague, and the metrics were not rigorously derived. van der Bij et al. (2013) suggest that the following practices make OSH ‘work’: “be open”; “make the design general enough”; “use standards and contribute to them”; and “be complete: from design to production test and drivers”. However, these suggestions are limited in that they are derived from the experiences of the authors who are from the same organisation and only develop open source electronics hardware.

In summary, while some work has been done on standardising documentation or describing best practices to produce ‘impact’, there is little work directly studying which features characterise the success of OSH projects in terms of both process and product. In the next sections, we highlight the gap in the literature which our study aims to fulfil, through the presentation of selected literature on success in OSS development (section Comparison with OSS development) and closed source NPD (section Comparison with closed source NPD).

Comparison with OSS development

Open source development has its origins in software, before its more recent expansion into hardware (Bonvoisin et al. 2018). What contributes to OSS project success has received significant attention, while this is not the case for OSH. Aksulu & Wade (2010) highlight studies that investigate determinants of OSS development success, and the potential relationships between them. Crowston, Annabi & Howison (2003) describes the development of success factors in OSS through literature reviews and practitioner opinion, and later investigate relationships between different success factors (Crowston et al. 2004). The Core Infrastructure Initiative (n.d.), a Linux Foundation project, has created a ‘best practices criteria’ self-certification badge programme to help OSS projects employ practices that relate to producing higher quality software (which relates to success). Examples of best practices include having a bug reporting process and using a publicly readable repository for storing files which enables version control. ¹³ Such practices could also be relevant in OSH projects, suggesting merit in comparing the two. Raasch (2011) suggested that when more practical applications of open design proliferate, research can illuminate the differences between OSH and OSS development – in this sense our study is timely.

While both OSH and OSS projects result in products with which a user interacts, there are substantial differences between hardware and software which influence the development process (Dai et al. 2020). Hardware is physical objects which are difficult and costly to change by the producer after manufacturing and distribution to end users, whereas software is flexible with newer iterations able to propagate with relative ease via software updates. Also, hardware development is more complex than software development as the former involves more considerations such as manufacturing, tooling, supply chain management. These discrepancies suggest differences in what constitutes success in OSH projects compared to OSS projects.

Crowston et al. (2003) identify a list of what they call ‘success measures’, characteristics of a project which influence how successful it is. Other studies on OSS investigate only a few specific characteristics of projects, for example, Sen, Singh & Borle (2012) investigate the number of developers and its determinants, while Midha & Palvia (2012) explore project popularity and developer activity. The seminal study of Crowston et al. (2003) on OSS project success is conceptually similar to our study and is the most appropriate point of comparison for our work as it focuses on the project level and success in general, rather than one or two specific project success characteristics.

Comparison with closed source NPD

Closed source value-capture mechanisms in the NPD literature revolve around restricting product design through patents and secrecy (James, Leiblein & Lu 2013), while OSH projects share their designs publicly, allow reuse via modification and/or duplication, and are characterised by transparency. In addition, in closed source NPD, a company tends to keep the design process exclusive to its employees, while, in contrast, certain OSH projects accept and encourage external participation. Furthermore, the motivations of participants and project organisations are likely to be different between closed source and open source development, which could impact project success. For example, in OSS projects, some people contribute not for financial gain, but to improve their skills (Hars & Ou 2001) – as is also the case in OSH (Hausberg & Spaeth 2020). A study on organisations based on OSH found that they are motivated not just by technological (e.g., standardisation), economic (e.g., research and development cost reduction) and product-based reasons (e.g., distribution permission), but also intrinsic factors such as personal satisfaction, altruism, hacker ethic and reciprocity (Li et al. 2017).

These differences in the development process and participant motivations could translate into a different view of what a successful project in each mode of development looks like. However, despite the contrasts outlined above, we expect that some insights from closed source NPD project management literature on project success, and some best practices, would apply to OSH projects.

Some project management literature on closed source NPD discusses success at the company level (Cooper & Kleinschmidt 1995) such as strategic success in innovation. However, our study is focused on what constitutes success within a project and comparisons are made to literature at this level.

In project management, the traditional way of evaluating project success is through the ‘triple constraint’, also known as the ‘iron triangle’, which contains three key dimensions: time, cost, and quality (Atkinson 1999). These dimensions relate to whether the project was completed on or ahead of time; within or under budget; and at the expected or higher quality. Usually, trade-offs occur between these dimensions.

Instead of the simplistic iron triangle, Shenhar & Dvir (2007) suggest five main dimensions of project success: efficiency, impact on the customer, impact on the team, business and direct success, and preparation for the future. Dvir & Shenhar (2011) later identified seven characteristics of successful projects, namely (i) they create competitive advantage and stakeholder value, (ii) a long time was taken to define them: creating a strong vision, clear need and choosing the most suitable execution approach, (iii) they create revolutionary project culture, (iv) they have highly qualified project leaders who are supported by top management, (v) they maximise the use of existing knowledge, often in cooperation with outside organisations, (vi) they have integrated development teams which are adaptive and have quick problem-solving skills and (vii) they have teams with ‘strong sense of partnership and pride’.

The closed source NPD literature is vast, with hundreds of papers and books written on the topic. For our study, we narrowed down the literature to only highly cited works that focus on NPD project-level success and where the descriptions of success characteristics are at an equivalent level of granularity to our dataset. As such, in section Comparison with NPD project management literature, we compare the results with the iron triangle (Atkinson 1999), the five dimensions of project success (Shenhar & Dvir 2007) and the seven characteristics of successful projects (Dvir & Shenhar 2011).

1.2. Significance

To summarise, there is a lack of studies examining success in OSH projects. To our knowledge, our research is the first to directly survey OSH practitioners with the aim of deriving common themes on what is considered success at the project level in terms of both process and product. We compare our findings to those in the OSS and NPD literature and identify success characteristics unique to OSH projects. This work is not only useful for furthering the study of OSH projects, but can also inform the OSS community or even closed source NPD.

2. Method

The first objective of the study, understanding success in OSH projects, was addressed by qualitative analysis of OSH practitioner responses to an open-ended question survey. Their opinions were used to identify the success characteristics of such projects and potential metrics for measuring success. The second objective, identifying aspects of success that are uniquely important to OSH projects, was fulfilled by comparing the results with selected relevant literature.

2.1. Survey design

Conducting a survey with open-ended questions is an effective method for collecting people’s opinions and experiences. To identify the characteristics of successful OSH projects, a written survey was designed and conducted to extract them from the experiences of practitioners.

The survey collected opinions on success factors, potential success metrics and essential practices in OSH projects. In combination, these would give a characterisation of project success in the context of OSH development, the main aim of this study.

The first round of the survey took place in February 2020 at an in-person academic workshop ¹⁴ focusing on OSH, where the respondents individually wrote down answers to the questions in physical (paper) format. Since most of the participants of that event were academics, a second round of the survey was conducted in digital format using an online survey tool, to reach a broader audience. This was disseminated through social media channels related to OSH, for example, the Twitter hashtags #opensourcehardware and #opensource as well as one of the author’s Twitter profile, who has a following of OSH practitioners and researchers from a variety of backgrounds such as designers, scientists, mechanical and software engineers; institutions such as OSH electronics manufacturers, distributors and collectives for developing collaborative solutions using OSH; and projects developing various types of hardware. The survey was live from 12 February to 30 April 2020. To screen for each respondent’s experience with OSH projects, they were asked to indicate whether they had participated in none, one or multiple OSH projects. They also provided their names and emails.

The following three open-ended questions were asked, each followed by a blank text box in which the respondents could write their answers.

1. (i) What does OSH project success mean to you? that is, examples of success factors (relates to RQ1)

2. (ii) What are some potential metrics for OSH project success? that is, what could be used to measure success (relates to RQ2)

3. (iii) What practices do you consider essential to successful OSH projects? that is, activities, artefacts (relates to RQ1)

2.2. Survey responses and demographics

We obtained 30 written survey responses: 10 responses from attendees of the academic workshop on OSH (30 participants at the workshop in total, therefore 33% response rate) and 20 responses via the online version. The responses varied in length, from some with short, bullet-point answers and others with long paragraphs of text. According to Mason (2010), the sample size is satisfactory for saturation. We also observed repetition in the data, which is demonstrated by the number of respondents who talked about each success characteristic (shown in Section 3). This also indicates data saturation.

The demographic of respondents can be described as follows: 8 had participated in one OSH project; 18 had participated in more than one and 4 had participated in none, but had research experience on the topic, or had the intention of publishing their hardware designs under an open source licence.

2.3. Data analysis and validation

The chosen data analysis method for the survey was thematic analysis, which involved coding the data set without a preexisting framework. This was done to place a focus on the informants (Gioia, Corley & Hamilton 2013), without imposing any preexisting ideas about success from the literature. Consequently, the themes relating to the success characteristics of OSH projects are as close as possible to the data itself, thus reducing bias. The analysis was conducted using the qualitative data analysis software NVivo 12.

Certain practices can lead to success factors through their cause-and-effect relationship. In addition, metrics can measure practices and success factors. This logical relationship between success factors, practices and metrics, combined with the fact that the respondents often did not make a distinction between them in their responses, lead to the responses to the survey questions being treated as one dataset during the analysis. This allowed the distillation of key themes from the dataset, with a large number of responses coded in each. This then enabled the results to be consolidated into the characteristics of successful OSH projects, and a list of metrics associated with them (see Section 3).

The generation of themes is a key feature of qualitative research and is dependent on the depth of understanding of the researcher. This is subsequently influenced by the researcher’s familiarity with the data sets and the research topic (Holton & Walsh 2017). Therefore, the coding was conducted iteratively, which increased this depth of understanding through the data analysis process. This also ensured that all the themes were captured, errors were reduced, and a rich description of the themes was achieved.

Where appropriate, in vivo codes ¹⁵ were used to stay close to the original data. Initial, intermediate and advanced coding were used, with increasing familiarity with the data. Initial coding involved basic fracturing of the data, intermediate coding involved grouping of the codes and transformation into themes, while advanced coding involved abstracting the highest-level themes, that is, characteristics (Chun Tie, Birks & Francis 2019).

The coding was primarily performed by a single researcher. To ensure validity, their coding was compared to that of two other senior researchers. The coding was tested both in breadth (the success characteristics) and in depth (the themes within one of the characteristics). The results of the test were calculated in percentage agreement (Caro et al. 1979) using equation (1). Percentage agreement is a frequently used metric for intercoder reliability tests using nominal data and was used in similar research such as that of Crowston et al. (2003). Agreement above 70% was achieved, which is considered sufficient to demonstrate the reliability and validity of the coding framework (Multon & Coleman 2018).

(1) $$ \mathrm{Agreement}\;\left[\%\right]=\frac{\mathrm{number}\ \mathrm{of}\ \mathrm{agreements}}{\mathrm{number}\ \mathrm{of}\ \mathrm{agreements}+\mathrm{number}\ \mathrm{of}\ \mathrm{disagreements}}\times 100\% $$

The outcome of the analysis includes several characteristics of successful projects and metrics for measuring some of them. These were grouped into three top-level themes: value creation, quality of output and project processes.

3. Results: characteristics of successful OSH projects

From the thematic analysis of the survey responses emerged three different but related themes regarding what characterises successful OSH projects: value creation, quality of output and effective processes. These themes influence each other: processes can influence quality of the output, and the quality of the output can influence value creation. This is summarised in Figure 1.

Figure 1. Relationships between the three themes identified from the thematic analysis of the survey responses.

In Sections 3.1–3.3, we describe these three themes through the insights derived from the data, delving into detail about what characterises successful OSH projects within each theme. In Section 3.4, we summarise the characteristics of successful OSH projects in the form of a table and provide suggestions for corresponding success metrics based on the data.

3.1. Successful OSH projects create value

This section presents the results from the survey responses which relate success to creating value, with 29 responses coded in this theme. Value refers to benefits, that is, positive outcomes or things of perceived importance. The respondents believe that successful projects create value to contributors, users, other projects and society. They also generate business activity and are sustainable over time. Popularity and a good reputation can indicate that they create value. Respondents also mentioned that popularity and reputation can be demonstrated by the ranking of projects on search engines; the number of projects, documentation, and scientific paper citations; the number of views and downloads of project documentation; the number of followers/interested people; and the presence of project communities with a high level of activity, for example, frequent participation in community forums. The following sections describe the types of value creation which were identified from the survey data.

Successful OSH projects create value to people and other projects

All 29 respondents, whose answers were coded in the top-level theme (Section 3.1), believe that successful projects create value specifically to people and other projects, with the majority referring to a large and vibrant community around a project to be indicative of success.

Successful OSH projects create value to contributors by way of personal gratification through “getting acceptance” by a community of users and satisfaction through creating something useful for others. They also generate value to contributors by giving them career impact, such as academic impact from paper publications and citation rates, as well as progression and development within the projects. As a result, contributors are motivated, interested and engaged in the projects, demonstrating long and continuous contribution. A potential metric for this is the number of third-party contributions, that is, contributions from people outside the core team of originators. Additionally, by creating value to contributors, projects can become more attractive to new ones, which could be indicated by the number of people who want to contribute – for example, by counting the number of forks of a project repository.

Successful OSH projects provide value to their users, which could be assessed by measuring how many people need the hardware those projects develop, that is, the market size for that hardware. In addition, the hardware produced by successful projects is helpful and useful to its users, is used and retained for a long time, while also being used in creative ways that were potentially not envisioned by the originators. Creating value to users can be indicated by their level of satisfaction with the hardware; their level of interest in the project; a high level of use, which could be measured by the number of units in use; the number of users (including those who built the hardware themselves); and a diverse user community (particularly including groups who did not have access to that technology before using the OSH).

Successful OSH projects create value to other projects. Several respondents believe that successful projects provided a basis for derivative projects and hardware, so the presence of such derivatives or ‘remixes’ is an indicator of success. The number of derivatives as well as ‘successful derivatives’ (as stated by the respondents) could be metrics of success for projects.

Successful OSH projects create value to society by contributing to “moving the state of the art forward incrementally” in technology, science and public health. They also allow others to learn, and they contribute to improving access to knowledge.

Successful OSH projects generate commercial value

The generation of business activity was identified as a characteristic of successful OSH projects, with eight respondents referring to it. It was stated that business aspects of projects should be “fostered and encouraged” in OSH. There were references in the data relating success to having a sustainable business; enabling commercial use through a relevant licence; selling hardware units or kits which are easily accessible; and generating revenue and profit.

Financial gain in the form of revenue and profit indicates a successful business and thus a successful project. It is closely related to the number of units sold, which could be a metric of success. Having independent vendors [other than the originator(s)] making and selling the hardware or its variants, as well as units selling well on the market for several years also indicate success.

Successful OSH projects create value sustainably

Seven respondents referred to project sustainability as being important to success. Project sustainability means that project activity could continue without the originator(s). Sustainability could be demonstrated by having funds available to conduct project operations, or actively planning for continuity of the project. A specific indicator of project sustainability mentioned in the data was the ‘bus factor’. The ‘bus factor’ indicates how many people would have to step down from the project (metaphorically ‘be hit by a bus’) for the project to be unable to continue (Cosentino, Izquierdo & Cabot 2015). Project sustainability is intrinsically linked to those that have a sustainable business. These are projects in which the business activity can continue and be maintained over time at the present level or higher.

Successful OSH projects create value to the open source movement

One respondent believes that successful projects contribute to the goals set in the GOSH Roadmap (Global Open Science Hardware 2018) whose aim is to make open science hardware ubiquitous by 2025. This characteristic is thus only applicable to OSH primarily designed for scientific applications. However, some of the goals could possibly apply to other types of OSH. This includes creating financial support structures for open science hardware, as well as preparing guidelines for different stakeholders (e.g., for compliance, licencing and documentation).

3.2. Successful OSH projects create high-quality outputs

A total of 27 respondents believe that successful OSH projects produce high-quality outputs in the form of hardware and documentation. The two are often related to each other. Some characteristics identified within this section relate to features relevant to definitions of OSH projects, such as that of the Open Source Hardware Association (2018). However, there is a degree to which these features can be implemented, which the respondents believe relates to project success, hence they are included in the results.

Successful OSH projects create high-quality hardware

Hardware quality, referred to by 16 respondents appears to be important to success. The “quality of the initial contribution” was suggested as an indicator of success. Successful projects create hardware which is performant and highly accessible, reproducible and modifiable. Their designs are also characterised by high transparency.

Successful OSH projects develop highly accessible hardware. Open standards and widely available tools are used as much as possible for production (e.g., manufacturing and assembly).

At least a prototype is available, and hardware units are being sold and easily accessible. The design and development of the hardware have proceeded enough to produce at least a prototype, which can be either made by individuals or bought. Ideally, completed units and/or kits are available for sale, and access to them is easy.

Successful OSH projects develop highly reproducible/replicable hardware. Replicability relates to whether external people can build the OSH using the documentation and raw materials. This can be demonstrated by the presence of individuals external to the projects who have built a working version of the hardware. The respondents mentioned ease of replicability in particular, which could be influenced by the quality of documentation (see section Successful OSH projects create high-quality documentation) as well as the availability of raw materials in the location of the person reproducing it.

Successful projects develop highly modifiable hardware. The hardware can be modified and adapted. This could be demonstrated by having evidence of others modifying the hardware to suit their unique purposes (e.g., by changing dimensions, materials and colours) or by adding new features (e.g., creating extensions and add-ons). The level of modifiability is influenced by a variety of factors including the presence of editable documentation (section Successful OSH projects create high-quality documentation).

Successful OSH projects develop performant hardware. When asked about what makes a successful OSH project, multiple respondents answered with a variant of “does [the hardware] work?” According to a specific respondent, a milestone is when the hardware becomes operational to relevant standards. The hardware must also be able to perform its intended function and have reliable performance.

Successful projects create highly transparent hardware designs. This could be demonstrated by projects selecting the most suitable open source licence for the projects and the hardware. Successful projects also fully disclose their designs with sufficient detail to enable any person with the relevant skills to build the associated hardware artefact. They further increase the level of transparency by ensuring it is easy for someone to build the hardware and understand how it works. This additionally contributes to the levels of accessibility and replicability.

Successful OSH projects develop hardware that solves a problem/fulfils a need and offers advantages over alternative products. The hardware “scratches an itch”, that is, solves a problem or fulfils a need of the user. Examples of this might be by providing a feature advantage over other products; giving them access to technology previously unattainable to them; offering a better-quality output; fulfilling a need or providing features that did not previously exist; or being more affordable than other offerings.

Successful OSH projects create high-quality documentation

Documentation quality is also important to success, with 25 respondents referring to it.

Successful OSH projects ensure the hardware source is highly accessible. The documentation is published on publicly accessible platforms such as GitHub ¹⁶ or GitLab ¹⁷ (commonly used version control repository-hosting platforms for open source projects) and is easy to find and download. The hardware source is also highly accessible in the sense that design and documentation files use open file formats, therefore not requiring the use of closed source software. The level of accessibility of hardware documentation can influence replicability and modifiability.

Successful OSH projects create documentation that is complete and has broad coverage. All the necessary documentation types are present, for at least a prototype of the hardware, such as BoMs, CAD files, design process documentation and user manuals. These influence the level of hardware replicability. Lessons learnt are tracked and could be captured in one or more documents. Such documents contain a log of the lessons which have the potential to be carried over to future or other parallel projects. These lessons could be technical or organisational. Successful OSH projects also have media and scientific publications. One survey respondent commented that the communication skills demonstrated in documentation could affect the level of usage of a project and its hardware.

Successful OSH projects create highly editable documentation. This means that people can easily make changes to it, which in turn increases the modifiability of the hardware.

3.3. Successful OSH projects have effective processes

This section presents the results from the survey which relate to the activities and processes that are part of successful OSH projects. The main finding was that successful projects have high process openness and follow product development, project, community and business management good practices. They are active, transparent, and committed to openness by sharing as much information as possible. Twenty-eight respondents referred to the different project activities facilitating success coded in this theme.

Successful OSH projects follow good product design and development practice

Seventeen respondents believe that good product design and development practice is important for success in OSH projects.

Successful OSH projects move through product development stages rapidly. This indicates a high level of activity. Certain respondents mentioned that successful projects have moved beyond the ideation stage: they are ready for use and are being manufactured and easily accessible.

Successful OSH projects develop hardware using good design practice. The survey respondents think good practice includes ensuring backwards compatibility of hardware versions and software; releasing a first version which is a minimum viable product (enabling the collection of feedback on the hardware); designing the hardware to be user-friendly and made of modular components; making prototypes; using CAD tools and using scientific reasoning for decision making. The number of design solutions considered as well as the number of design iterations was also mentioned as potential metrics of success relating to good design practice.

Successful OSH projects have design and development processes that enable product openness. The respondents think that successful projects use parametric design methods to facilitate customisation and enable modifiability, which in turn increases openness. They also mentioned that the availability of raw materials around the world should be considered by hardware designers to increase replicability. The ability to build the hardware using ‘everyday tools’ would also facilitate replicability.

Successful OSH projects develop hardware using user-centred design. The data showed that successful projects design their hardware with their users in mind.

Successful OSH projects have effective management and teamwork

Twenty-two respondents believe that effective project management and teamwork are important for success in OSH projects.

Successful OSH projects are managed effectively. They demonstrate effective project management by using version control software (platforms enabling the recording of file changes over time); having traceable contributions; following clear aims; having ‘good governance’; and being actively maintained. The latter could be measured using the time taken to close issues that are flagged up in the project repositories.

Successful OSH projects effectively engage and manage their user and follower communities. They foster a vibrant community of users and followers, make an effort to build a user and follower community, and exhibit frequent and clear communication and support. For example, a successful project might have a website where the project is introduced and explained, and an online forum for community participation and support. Successful projects additionally engage their user and follower community by participating in community events such as workshops.

Successful OSH projects engage potential contributors, and existing ones work together effectively. They actively engage contributors by making contributions easy. They do so by documenting the design, the decision-making process and the lessons learnt, which assists future work. Successful projects document early on and have contribution policies and structured knowledge bases for contributors. They also adopt contributed modifications. A successful project’s contributors share similar expertise, contribute in diverse ways (e.g., designing and bug fixing), and demonstrate effective collaboration, co-creation, and teamwork. The number of people who contribute to a project, including the presence of commercial/industry contributors could indicate success.

Successful OSH projects are committed to openness

The respondents believe that successful OSH projects engage ‘openly and transparently’ and fully disclose information. Thirteen respondents believe that successful projects must be committed to openness.

Successful OSH projects develop hardware and documentation using an open source toolchain. This relates to the use of OSS for creating CAD files, manufacturing files and any associated documentation and software.

Successful OSH projects track lessons learnt and publicly share them, indicating a level of knowledge management and a means to transfer knowledge across projects.

Successful OSH projects enable commercial use. They do so by publishing their source files with an open source licence that enables commercial use.

Successful OSH projects are committed to openness even on occasions where it might be opposed by certain external forces. Respondents identified the need for a commitment to openness for success as some had experienced some barriers to being open source, such as ‘commercial expectations’ and ‘cost’. They may also have been referring to cases such as MakerBot ¹⁸ which changed to closed source after initially being open source. Additional commitment to openness seen in successful projects is the use of OSS to conduct their everyday activities.

3.4. Summary of results

In Tables 1, 2 and 3, we provide a summary of the characteristics of successful OSH projects described in Sections 3.1–3.3, and give a list of metrics for measuring progress towards them. We have identified a total of 101 metrics. Most of the metrics are uniquely linked to each characteristic, however, two of them (presence of commercial use licence and presence of lessons learnt log) relate to more than one.

Table 1. Summary of the project characteristic ‘Successful OSH projects create value’, along with potential metrics.

Table 2. Summary of the project characteristic ‘Successful OSH projects create high-quality outputs’, along with potential metrics.

Table 3. Summary of the project characteristic ‘Successful OSH projects have effective processes’, along with potential metrics.

4. Discussion of findings

In this section, the characteristics of successful OSH projects presented in Section 3 are discussed. In Section 4.1, we compare the results with OSS literature and NPD project management literature before analysing aspects of success uniquely important to OSH in Section 4.2.

4.1. Comparison of findings with selected relevant literature

This section presents a comparison of the results presented in Section 3 with OSS literature (section Comparison with OSS literature) and NPD project management literature (section Comparison with NPD project management literature).

Comparison with OSS literature

There is a similarity between our results on OSH success and literature on OSS success. The results confirm the nonsoftware related ‘success measures’ identified by Crowston et al. (2003), such as number of contributors/developers, level of activity, bug fixing performance, number of downloads, design/code reuse, user and project member positive outcomes (satisfaction, reputation, etc.), process quality and software/hardware quality. Table 4 summarises that comparison.

Table 4. Comparison of the results of the presented study with that of Crowston et al. (2003) on OSS project success

When compared to the Core Infrastructure Initiative (n.d.) Free/Libre and Open Source Software (FLOSS) Best Practices Criteria which are used as part of a certification programme, we observed that apart from software-specific practices (code analysis, software security, etc.), best practices they suggest are confirmed in the results of the here presented survey. Examples of best practices common to both OSS and OSH include having an open source licence; having a defined governance model; having up-to-date documentation; having a high ‘bus factor’ (they suggest a minimum of 2 for their highest-level ‘Gold’); using distributed version control such as Git; and using an issue tracker for tracking different issues or bugs that may arise. This certification programme also provides some specific practices which relate to some of the more abstract characteristics identified. For example, they suggest that the project clearly identifies small tasks which could be undertaken by new or casual contributors. This relates to section Successful OSH projects have effective management and teamwork, where attracting new contributors is discussed.

There were some differences between our results and Crowston et al. (2003), such as their finding of ‘varied developers’ as a success measure while we found ‘developers sharing similar expertise’. This contradiction should be further investigated. Another difference was that Crowston et al. (2003) determined that negative attention towards the project could be beneficial, but we only found references to having a good reputation and positive attention in our study. Some differences include that the Core Infrastructure Initiative Best Practice Criteria also include certain practices which did not appear in the OSH survey. Examples of these include considering accessibility requirements for people with disabilities; requiring cryptographic two-factor authentication for changing the central repository or accessing sensitive data; defining key roles and responsibilities in a publicly shared document; linking tasks and people to those roles; and finally, having a community code of conduct.

Themes emerging from our study which were not identified by Crowston et al. (2003) include creating wider social impact (e.g., providing a product/tool that was previously inaccessible to certain groups of people); active attempts by the project to engage and grow the community around it; having good governance; and being sustainable (in terms of continuity of project and/or business). Furthermore, our study provides more detail and depth into certain themes. For example, we not only identified the importance placed on documentation quality, but also specific practices that affect it. It is, however, notable that while documentation is important to success, in practice OSH participants ‘are not motivated to document’ (Dai et al. 2020).

Comparison with NPD project management literature

When it comes to the iron triangle (time, cost and quality), our results refer to time in the sense of having rapid development, but no indication was given about completing a project ‘on time’. Instead, we observed an underlying assumption in OSH product development that the project will be ongoing. Even if only a first version of the hardware will be developed in the project, the assumption is that eventually more iterations would follow. Project cost only appeared in the survey results in the context of having secured ‘enough’ funds for the project to continue performing its operations. Quality appeared in the survey results in terms of hardware and documentation, but also indirectly in the form of quality of the employed processes.

Four of the five dimensions of project success suggested by Shenhar & Dvir (2007) are confirmed in our study, namely: impact on the customer (the users), impact on the team (the contributors), business and direct success and preparation for the future. These are reported in Section 3.1 on value creation. While the fifth dimension, efficiency, is not explicitly evident in our results, it could be linked to the project processes (reported in Section 3.3). Our work adds depth to these dimensions by offering potential ways of measuring them in OSH projects.

In Table 5, we make a comparison with the seven characteristics of highly successful projects identified by Dvir & Shenhar (2011).

Table 5. Comparison of OSH project success characteristics with Dvir & Shenhar’s (2011) project success characteristics

4.2. Open source hardware project success

This section presents a discussion of the themes uniquely important to OSH which emerged through the survey results. Section Product openness contributes to success discusses product openness; section Process openness contributes to success considers the contribution of process openness to success; section Ethical, societal and political motivations expands on the ethical, societal and political motivations in OSH projects; section Business and Sustainability details the importance of business and sustainability, while section Peer-reviewed publications addresses peer-reviewed publications.

Product openness contributes to success

The success characteristics identified in this study confirm the definition of OSH given by the Open Source Hardware Association (2018) as well as the four rights of OSH given by DIN SPEC 3105 which are essential to hardware being defined as OSH: that anyone can study, modify, make and distribute it. These definitions only refer to product openness. Transparency, full disclosure and an OSH licence would allow the four freedoms expressed in the definitions. However, there are aspects of projects beyond a licence that are necessary in order to exercise the freedoms. For example, our results show that the hardware should be easily accessible for purchase from somewhere, which is not present in these.

The results of our study show that hardware sales by independent vendors different from the originator(s) can also be a sign of success. The existence of such vendors could indicate demand for the product. As such, other people see value in selling it, because it can generate a profit or other value.

Process openness contributes to success

Even though – according to the OSH definition (Open Source Hardware Association 2018) – only product openness is required for hardware to be termed open source, our results also identify having process openness to be a characteristic of successful OSH projects. This confirms all eight of the Open-o-Meter criteria identified by Bonvoisin & Mies (2018), namely the presence and use of an OSH-compatible licence, design files, BoM, assembly instructions, original files, a version control system, a contribution guide and an issue tracking system. Our results also hint towards additional process openness criteria, for example, presence of online forums and chats.

Ethical, societal and political motivations

The survey findings also confirm some already-known ethical, political and societal sentiments which often underpin people’s motivations for participating in and advocating for, open source development. We observe responses mentioning that projects following an open source ‘path’ might not necessarily be the cheapest – that is, financial sacrifices may be made for the ‘higher good’ of remaining open. Even though it is unclear in the data how this may manifest, it underpins a sentiment of making sacrifices if needed to maintain open source status.

The democratisation of knowledge was a recurrent theme in the responses. This indicates a sentiment of sharing information and knowledge with others without discrimination, that is, the inherently political notion of equal rights for access to knowledge.

Multiple survey responses made references of wider social impact as being a characteristic of project success, in that the project creates value to science and society. Some quotes from the survey include: “giv[es] access to tools usually out of reach to the less fortunate”, “enables learning”, “helps democratise knowledge”, “citizen science movement”, and so forth. We observed a notion of accountability on OSH projects to be of value to human lives and the evolution of society. The respondents believe that for these projects to be successful, they must somehow contribute to this ‘higher cause’ of bringing about positive scientific and social change. Examples mentioned include allowing knowledge to be democratised and disseminated to all those who need it, without discrimination; enabling access to tools that were previously not available to certain communities of people; and more generally contributing to the development of science and technology.

These factors point to one of the core principles of open source development, which is accessibility. While Balka et al. (2014) define accessibility in terms of a person being able to participate in the product development process, in our study we found that accessibility can take additional meanings. Our data gave examples of accessibility such as: access to the original ‘source’ of the product; access to materials needed to make the product; access to an assembled or do-it-yourself (DIY) kit of the hardware; but also access to a knowledge or capability – notions whose value to individuals and society are less tangible or measurable.

Democratisation of knowledge

When referring to democratising knowledge, we denote the spreading of knowledge amongst all people, without discrimination, not just limiting it to those who have certain privileges. A cornerstone of the democratisation of knowledge is therefore access to information. OSH – and in general the open source movement– are inherently contributing to the democratisation of knowledge by their nature itself– the blueprints of the products are openly shared, sometimes along with the design process too. Even projects on the lower end of the ‘openness scale’ (see Open-o-Meter by Bonvoisin & Mies 2018), still provide a certain contribution to the democratisation of knowledge, in comparison to closed source hardware developed through conventional product development. One might argue that the technical features of some closed source hardware are publicly shared if it is patented. However, patents describe little beyond the working principle(s) and rarely provide details on materials, specific components, dimensions or manufacture. While patents might provide some access to knowledge, they prohibit using that knowledge in a meaningful way without obtaining a proprietary licence from the patent holder.

Survey respondents believe that a successful project might be characterised by its contribution to the democratisation of knowledge. From this, it is possible to hypothesise that the extent of its contribution to the democratisation of knowledge, relates to the extent of the project’s success, and is worthy of future study.

Business and sustainability

Conducting business activity was identified as a characteristic of successful OSH projects in this study. Commercial success validates the product itself, proving the value of the hardware, as well as the viability of OSH for conducting a profitable business.

The sustainability of both the project and any associated business was a theme that emerged from the survey responses. Sustainability in this context means the ability of the project to continue conducting its operations and activities “beyond the lifetime of the creator”. Sustainability is influenced by how much knowledge is available; how well that knowledge is shared; the ‘bus factor’; and funds. Funding emerged as an issue because it influences how much a project can do and how well it could sustain itself in the future.

Scholars have proposed creating sustainable value in OSH (Moritz et al. 2017). Research has also proposed using OSH as a business model for companies (Li & Seering 2019), with the option of moving away from if they wanted, rather than basing the company around the OSH product(s). They advise companies using OSH to make the OSH development model more sustainable such as: develop a strong brand; have fast innovation with the assistance of the community; and use the knowledge and experience gained through what they call the “open source stage”; and then produce closed source associated hardware and/or extensions. The latter, however, may be perceived by some to be against the open source ethos and this has been specifically pointed out in the survey responses of the present study. Companies who have done this have indeed attracted criticism. For example, MakerBot who released its first version as OSH, and was itself based on the OSH Rep Rap 3D printer, received such criticism (Brow 2012; Hall 2016). Pearce (2017) does not consider OSH as a singular business model, and instead outlines a variety of business models that could be used in an OSH project, depending on the audience, for example, selling self-assembly kits of the hardware, selling preassembled hardware units, selling a service based on the hardware.

Our findings show that conducting business and being sustainable over time are important factors by which the success of a project can be evaluated, and thus relevant metrics and indicators could be used to assess them.

Peer-reviewed publications

Peer-reviewed publications are especially important in the academic OSH community as the number of which is a metric that influences an academic’s career and thereby creates value to academic contributors. It also gives a certain prestige and officiality to the associated hardware if an extended form of its documentation is published in an academic journal. A few OSH-focused journals exist which accept submissions for OSH designs.

4.3. Limitations and future work

This study provides insights into characteristics of successful OSH projects, some preliminary best practices and metrics for measuring success. Further studies could investigate creating tailored best practice suggestions for OSH projects based on their unique contextual factors, such as the type of product being developed. These could then form the basis of a guideline for helping OSH projects steer themselves towards success and could also inform the development of standards.

Dashboards, graphical user interfaces used for giving visualisations of key performance indicators for projects, are increasingly used on project hosting websites such as GitHub to give visitors and developers at-a-glance information about the status of each project. Dashboards may help potential contributors select projects that they are interested in. The metrics we identify could be implemented on such dashboards on online OSH project repositories, and they can then be used in conjunction with suggestions for ways to improve the scores on those metrics. In this way, the outcomes from OSH projects could be improved.

The existing data set and insights from this study could also be further analysed to produce a draft framework for the relationships between value creation, quality of output and project processes observed in successful OSH projects. Conversely, this framework could then also be used as a basis for analysing how and why OSH projects fail.

Adaptive project management is a method that involves adapting the style of managing the project based on certain variables (Shenhar & Dvir 2007). While we expect adaptive project management to be applicable to OSH development, further studies could investigate the relevant variables.

While our results provide a step forward in characterising success in OSH, it is important to highlight its limitations. The first of those relates to the sample used. Given the exploratory nature of the study, we used a qualitative approach with a sample of 30 individuals. While this approach enabled us to use rich insights for uncovering relevant themes in defining success and to reach saturation, we cannot claim the sample is representative of the entire population of OSH practitioners. Moreover, while we distributed the survey in person and online to practitioners, we cannot completely exclude bias due to self-selection. It is also possible that the dataset is biassed towards a certain group of OSH practitioners, for instance, those who only participate in projects which develop a certain type of hardware, for example, electronics. To mitigate the sampling limitations, future studies could collect a larger number of respondents through a wider range of platforms, as well as capture more information on the backgrounds of those respondents. The latter could also aid in discovering what success characteristics, practices and metrics are related to specific types of OSH projects.

Second, the sample does not allow us to draw conclusions on the relative importance of each of the themes identified, nor were any metrics used to objectively evaluate success in projects. Thirdly, while we focused on practitioners, we do not investigate the role or experience in OSH projects of the respondents (e.g., project initiators, contributors and end users). Future studies could identify the relative importance of the success characteristics in relation to OSH participants’ roles and levels of expertise. Furthermore, as described in Section 2.3, the answers to the survey questions were treated as one dataset. Further studies could research factors, practices and metrics for success individually in more depth, along with the relationships between them. Lastly, a quantitative research study measuring success in OSH projects could test the validity of our conclusions.

Despite these limitations, we believe that this study can provide useful insights both to OSH practitioners and scholars interested in understanding how to support the success of OSH projects. We also hope that this study will foster the discussion on the specific characteristics of the OSH community.

5. Conclusions

This study is a first step in characterising OSH project success and identifying success characteristics that are uniquely important to OSH development from the point of view of practitioners. Using thematic analysis on a dataset of written answers to open-ended survey questions given by OSH practitioners, we outline various characteristics of successful OSH projects through three high-level themes. Those themes are ‘successful projects create value’, ‘successful OSH projects create high-quality outputs’ and ‘successful projects have effective processes’.

We also suggest some practices for promoting success and metrics for measuring it which were indicated by the dataset. Furthermore, we contrast OSH success with success in OSS and NPD project management literature. This allowed us to present success characteristics that relate to OSH projects specifically. Examples include having process openness which brings about wider social impact; providing access to new knowledge; giving access to a tool/product/device previously unavailable to certain groups of people; and having business and project sustainability over time.

The insights from this study answered the research questions ‘What characteristics and practices are present in successful OSH projects?’ (RQ1) and ‘What metrics can be used to measure success in OSH projects?’ (RQ2) and fulfilled the objectives of understanding success in OSH projects and identifying success characteristics that are uniquely important to OSH development. Consequently, the results have implications for practitioners when planning and managing an OSH project, and provide a basis for future work for researchers studying factors leading to OSH success. This study can also help inform the creation of a success guideline for OSH projects.