The Combinator – a computer-based tool for creative idea generation based on a simulation approach

Abstract

Idea generation is significant in design, but coming up with creative ideas is often challenging. This paper presents a computer-based tool, called the Combinator, for assisting designers to produce creative ideas. The tool is developed based on an approach simulating aspects of human cognition in achieving combinational creativity. It can generate combinational prompts in text and image forms through combining unrelated ideas. A case study has been conducted to evaluate the Combinator. The study results indicate that the Combinator, in its current formulation, has assisted the tool users involved in the case study in improving the fluency of idea generation, as well as increasing the originality, usefulness, and flexibility of the ideas generated. The results also indicate that the tool could benefit its users in generating high-novelty and high-quality ideas effectively. The Combinator is considered to be beneficial in expanding the design space, increasing better idea occurrence, improving design space exploration, and enhancing the design success rate.

1 Introduction

Everyone can and does design to some extent, as design is a natural cognitive function of the human brain (Cross 2011). Design is identified as a primary driver of innovation, which can benefit a company’s business performance as well as its brand identity (Roper et al. 2016). It tends to benefit from the generation of alternative ideas. However, coming up with new and useful ideas is often challenging. Idea generation or ideation is the process of generating ideas in a design activity, and it is where a design concept begins. It essentially determines the type of design produced and the value of business performed. Idea generation plays a vital role in novel concept development and a key to success in business competition (Howard, Culley & Dekoninck 2011). Creativity is a key factor influencing whether an enterprise can achieve success. For instance, Dyson and Apple have achieved great success due to their creative products and organisation, whereas firms like Nokia had lost market share as they could not sustain their creative and business edge.

Recent research has indicated the relation between design, creativity, and business profit. Design can be described as a specific end to the deployment of creativity (Design Council 2011). Historically, the UK’s most innovative companies generate over 75% of their profits from products that did not exist 5 years previously (Cox 2005). Recently, the UK Design Council (2015) revealed that design contributed £71.7 billion in gross value added (GVA), equivalent to 7.2% of total GVA. This indicates that business activity is indirectly related to creativity via design and suggests that generating creative ideas is necessary to novel concept development and ultimately innovation (Howard et al. 2011). More ideas, especially creative ideas, are required to be developed for the needs of producing robust designs, and thereby promoting commerce.

However, generating creative ideas is difficult, due to problems such as lack of creative minds, project time pressure, numerous existing ideas, and limited relevant information. Childs (2014) has indicated that an enormous amount of information, the willingness and endeavour of producing ideas, and the capability of discarding irrelevant ideas are essential elements for generating creative ideas. Nevertheless, creativity is a notorious elusive phenomenon, often associated with human genius and serendipitous discovery. As a result, there has been an increasing interest in recent decades in investigating the factors that encourage creativity and exploring how it can be fostered.

A number of creativity tools have been developed to help designers invent new ideas. These tools do not actually generate ideas (Childs 2014), but stimulate the users by removing mental blocks and expanding searching space (Cross 2008). Some tools such as, brainstorming (Osborn 1979) and mind mapping (Buzan & Buzan 1996), are fairly easy to grasp but rely heavily on users’ experience and knowledge. Others, such as TRIZ (Altshuller 1984; Gadd 2011) are rather complex and difficult to master. Some, such as lateral thinking (De Bono 2016), are inconsistent with normal human thought processes. The information-processing theory of bio-inspired design developed by Goel et al. (2014) for composing new design concepts entails cross-domain analogies and problem–solution coevolution. The 77 design heuristics identified by Yilmaz et al. (2016) are intuitive for designers, but it is difficult to decide which heuristic to apply in a given design task. Moreover, Yan & Childs (2015) have indicated that different tools possess different characteristics and are suitable for diverse applications and personality attributes.

A number of computer tools have been developed to assist idea generation, such as the IDeAL system based on analogy and design patterns (Goel & Bhatta 2004), the virtual concept generation process (Taura et al. 2012), the matrix-based concept generation system (Kang & Tang 2013), and one based on chains of physical laws and complementary basic schemata (Zavbi, Fain & Rihtaršič 2013). Recently, Bacciotti, Borgianni & Rotini (2016) have illustrated a method, which has been implemented in a computer application, for supporting the idea generation in early engineering design phases. In this approach, concepts from two different dimensions are combined for identifying scenarios, in which the product can provide great benefits for customers, to provoke the creative process of the users. However, the tool users need to confront a protracted series of questions and stimuli, which might place the tool in boredom. Chakrabarti et al. (2017) have developed an analogical design tool, called Idea Inspire 3.0, to assist designers in ideation. The tool could retrieve systems from biological domains to assist idea generation to solve a given problem. However, user trainings are suggested before using the tool. Georgiev, Sumitani & Taura (2017) have proposed a methodology for creating new scenes for developing new products by synthesising existing scenes which belonged to different contexts. However, as a new scene is created based on a keyword, the new scene created might be unrelated to the existing scenes of the keyword.

In order to assist designers in generating creative ideas intuitively and effectively, without requiring very sophisticated design experience, design knowledge and user training, this research aims at developing a simple-to-learn and easy-to-use computer-based tool for idea generation, called the Combinator, based on a previous study (Han, Shi & Childs 2016). The current version of the Combinator is intended to assist novice as well as experienced product designers in inventing new ideas during early design phases, through combining familiar ideas due to a customised database containing ideas of product design. It has already been suggested that combining concepts or nouns can benefit creative idea generation at early design phases (Taura et al. 2007; Nagai, Taura & Mukai 2009). The basic algorithm of the tool is a simulation of aspects of the human mind and memory in performing combinational creativity (Boden 2004). The Combinator is suitable for various people as it matches with common human thought modes and can provide users with a relevant knowledge database. It is hoped that the tool will help both novice designers and experienced designers tackle the challenges of the fast-moving business market.

The next section reviews some aspects of combinational creativity and explores some basic models of how the human mind and memory functions. Section 3 demonstrates the Combinator’s basic algorithm and how the tool has been developed. Section 4 provides an evaluation of the Combinator through a case study. Finally, a general conclusion is presented in the last section.

2 Combinational creativity

2.1 Creativity

Creativity is defined as ‘the ability to imagine or invent something new of value’ (Childs et al. 2006), or ‘the production of novel, useful products’ (Mumford 2003). It is considered as a vital element in design, which initiates innovations, assists problem solving, and increases a firm’s market share (Sarkar & Chakrabarti 2011). Stewart & Simmons (2010) indicated that it is creativity which will drive business in the future.

Creativity is a fundamental feature of human intelligence (Cross 2011). The outputs of creativity can be objects, actions, or ideas, and these outputs are conceived to be novel, high in quality, and useful (Carruthers 2011). The creativity output arises from the combination of some essential mental capabilities, which is often the result of long periods of work with a series of mini-breakthroughs (Childs 2014). Most creative ideas can be characterised by the diverse thoughts from previous experience stored in memory elements (Liikkanen & Perttula 2010; Benedek et al. 2014).

2.2 Combinational creativity

Boden (2004) has proposed three approaches to achieve creative ideas in our human brain, which are combinational, exploratory and transformational creativity. Exploratory creativity includes the generation of creative ideas through exploring structured conceptual space (structured styles of thought), for instance, the different styles of coffee mugs. Transformational creativity involves transforming the conceptual space to produce transformative ideas in new structures, for example, the origination of aromatic chemistry by transforming string molecules to ring molecules. Combinational creativity, also known as conceptual combination, involves generating new ideas through exploring unfamiliar combinations of familiar ideas (Ward & Kolomyts 2010).

Combinational creativity can be achieved through forming associations between ideas that were previously only indirectly linked (Boden 2004). The ‘ideas’ to be combined can be concepts, words, images, sounds, and even more abstract ones such as, music styles, artistic genres, and so on (Ward & Kolomyts 2010). Combinational creativity is the easiest approach to creativity for human beings, as it is a natural feature of the human associative memory system (Boden 2009). Combining ideas could be the best way to fully use nowadays abundant information for producing creativity (Yang & Zhang 2016). Many people have used the term ‘combinational creativity’ to describe creativity, for example, Steve Jobs defined that ‘creativity is just connecting things’, Frigotto & Riccaboni (2011) indicated the nature of creativity is to combine, and Henriksen et al. (2014) described creativity as ‘the process of making alterations to, and new combinations with, pre-existing ideas and artifacts, to create something new’.

Combinational creativity has been used widely in creating various products. For instance, the well-known chocolate bar ‘Kit-Kat’ can be regarded as just a simple combination of chocolate and wafers; the toy ‘LEGO’ is a series of combinations of numerous different bricks; the ‘Dyson’ vacuum cleaner combines cyclone and ball with vacuum cleaner which provided a strong suction power and a steering mechanism, respectively; the ‘Apple Watch’ is a combination of watch and data device, albeit with a very sophisticated operating system.

Although combinational creativity is the easiest type of creativity for humans, it is challenging for computer-based implementation. Boden (2004) noted that combinational creativity requires a very rich store of knowledge, and the ability to form links of many different types. However, combinations of ideas might lead to a ‘combinational explosion’, of which it could take years to produce and evaluate all the possibilities (Simonton 2017). These are also some of the main reasons why acclaimed designers are usually rich in knowledge and experience. It is not difficult for computers to form combinations or even novel combinations of stored ideas, but it is extremely difficult for computers to generate valuable or interesting combinations and to recognise their values (Boden 2009). Compared with the human brain, it is difficult for computers to achieve the cornucopia of human associative memory. Moreover, a computer cannot naturally form links between ideas and evaluate them, while a human can naturally associate ideas of many different kinds during everyday thinking. Therefore, modelling combinational creativity on computational systems requires a rich knowledge database and methods of generating links comparable to a human. Some successful computer programs that involve combinational creativity are reported by Gastronaut (Butnariu & Veale 2006) and Portrait Robot (Augello et al. 2014).

In order to simulate aspects of the human brain in producing combinational creativity, understanding how the human mind and human memory work is essential. The following sections briefly illustrate the basic working principles of the human mind and memory in producing combinational ideas. In this study, human memory is considered as a knowledge database containing various ideas, while the human mind is regarded as an associative system that can form links among stored ideas.

2.3 Human memory – the knowledge database

Memory can be described as the ability to store, retain, and recall information (Shergill 2012). It is significant to designers, as an idea generation process involves controlled retrieval of existing information from memory (Liikkanen & Perttula 2010; Benedek et al. 2014). The multi-store memory model proposed by Atkinson & Shiffrin (1968) comprising long-term memory and short-term memory is often cited, and it is used to explain how human memory works in this study.

Short-term memory is a system that underpins the capacity to retain small amounts of materials over periods of a few seconds (Baddeley, Eysenck & Anderson 2009). Short-term memory is also known as working memory, and it is the place where information is manipulated and processed (Cowan 2008). Long-term memory stores information over long periods of time (Baddeley et al. 2009). Information from the outside world flows into short-term memory through sensory perceptions (such as visual and auditory), and then the information can flow into long-term memory through encoding. The information can be passed back into short-term memory by retrieval while cues from the outside world or ourselves are provided. Through refreshing (Vergauwe & Cowan 2015), the key information in short-term memory can be maintained for a relatively long time. An information flow diagram of the human memory system using this embodiment is shown in Figure 1.

Figure 1. Information flow diagram of a human memory system model.

Although human memory has extensive store capacity and duration, the performance on tasks involving long-term memory declines steadily through the adult years. It seems that human memory is less reliable than augmented memory (non-single human mental memory) (Baddeley et al. 2009). A number of augmented memories, for example, books, Internet, and professional memory augmentation systems such as Ubiquitous Memories (Kawamura et al. 2007), are available to help us store and recollect memory elements.

2.4 The human mind – the associative system

The human mind is widely considered as a product of natural selection and evolution (Bolhuis & Wynne 2009). Intuition is a fundamental facet of our thinking and used by us to solve all kinds of problems (Pinker 1999). Connectionism is a widely accepted approach for explaining how the human mind works, which is based upon highly interconnected processing units (Thomas & McClelland 2008). The human associative mind is a connectionist system, with information elements associated together in the mind through experience (Anderson & Bower 2014). Associative memory is an attribute of the mind’s associative power, which is the ability to learn and remember the relationships between different information. For instance, we recall a person’s face and other characteristics immediately when the known person’s name is mentioned (Anderson 1976; Suzuki 2005). Due to this ability, an enormous associative network involving information and relationships has been formed in the human long-term memory through learning and experiencing. Many cases of creativity depend on the mind’s associative power, especially combinational creativity (Boden 2004).

A semantic net is an artificial associative network in a graph structure that represents knowledge in relation patterns of interconnected nodes and links (Sowa 1992). It is a depiction of human associative memory as well as an associative model of cognition, in which each idea can be linked to some other relevant ideas and sometimes ‘irrelevant’ ideas. Combinational creativity, as defined, involves generating new ideas through associating unrelated ideas. Therefore, achieving combinational creativity can be regarded as forming new combinational links between unrelated or indirectly related ideas in a semantic net.

3 The Combinator

The Combinator is a piece of software developed for assisting novice designers as well as experienced designers in idea generation during the early phases of design. The tool is based on a simulation of aspects of the human brain in achieving combinational creativity. It can combine familiar ideas together for generating new combinational ones. The generated ideas are not just limited to text forms but also visual forms. The Combinator is aimed at supporting ideation while a high degree of problem specification has been reached. Nevertheless, the tool could also support idea generation at very early design phases while design objectives have not been set, as it can produce a variety of random ideas which could stimulate users’ creativity. Thereby, the tool could also be used to provide stimuli or inspirations for supporting designers in creative ideation at design contests. The tool is expected to be useful for young and novice designers, as the tool can deliver new ideas or inspirations which novice designers are lack of. It should also benefit experienced designers providing unfamiliar ideas as well as cues to recollect ideas. As a tool provoking users’ creative brain, the tool has a good aptitude in solving design problems and producing creative design ideas, especially in early design phases.

At the moment, the Combinator, in its current formula, is used to demonstrate its performance of supporting designers, especially novice designers, in producing creative ideas in an academic or teaching context. However, the Combinator is a flexible tool that could be adjusted to support different purposes by using different databases. For example, a large design firm could employ its own database for producing ‘tailored’ creative combinational ideas which are composed of its own products and techniques.

3.1 The basic algorithm of the Combinator

This section and the following two sections illustrate the novel approach used for developing the Combinator. This type of approach can be regarded as a new method for developing design tools, especially idea generation tools. In this study, the Combinator is aimed at ideation in the product design field. However, it is believed that the Combinator can be used in various fields, such as engineering, fashion, music, and art. Understanding how the human mind and memory work is crucial for simulating combinational creativity on computers. Memory is the place where structured information is stored, and the mind is related to how structured information and new links between unrelated information are formed. Therefore, in order to simulate combinational creativity, modelling a rich structured database and an idea linking system are necessary. The basic algorithm of the Combinator, is shown in Figure 2.

Figure 2. The basic algorithm of the Combinator. © Ji Han.

The algorithm starts with trawling and retrieving relevant user-defined information from the Internet using web crawlers, which is similar to a human in gaining knowledge from augmented memory. The information is then stored in the core database and linked with a semantic network after being analysed and processed by a natural language processing tool. This is a process simulating how knowledge is processed and related to each other in human memory. When a user input, which can be a design keyword or a desire, has been delivered to the Combinator module, a cue is produced for instructing the retrieval process from the Combinator database to the Combinator module. This is similar to how information in human long-term memory is retrieved and transferred into short-term memory. The ideas in the Combinator are linked together, thereby generating combinational ideas in text forms as well as visual forms. This imitates aspects of how the human mind forms links between ideas to generate combinational ideas.

As illustrated above, the basic algorithm of the Combinator conforms to natural features of the human mind and memory system in generating combinational creative ideas. Thereby, the natural feature of the algorithm has provided a method accommodating a human’s natural thinking pattern, which indicated simple-to-learn and easy-to-use characteristics.

3.2 Creating the Combinator database

Creating a human-like structured knowledge database is the initial stage, but not every piece of information is relevant to product design. Therefore, it is necessary to narrow down data elements and discard irrelevant elements in order to produce creative ideas (Childs 2014). The database can be simply generated via manual input, but this is time-consuming and capacity limited. This study takes advantage of natural language processing tools, information retrieval technology, and existing knowledge bases to produce the Combinator database.

A web crawler is a program that browses and retrieves specific resources and information from the Internet (Patil, Bhattacharjee & Ghosh 2014). This technology has been used for various purposes, such as data mining, information searching, web monitoring, and web archiving. It has been demonstrated that web crawlers can be used to mine design-related information for idea generation (Wang, Childs & Jiang 2013).

An open-source web crawler is used in this study for developing a text-based knowledge database. This allows users to define which domain or group of domains to explore, for example, what kind of data to extract and which data format to use. In order to narrow down the database in a product design field, the web crawler was programmed to retrieve only main body texts describing design-related products or ideas from a design website that covers the best in product design. Simultaneously with the crawling, the crawled texts were processed by a natural language processing tool, to extract requisite words. Texts such as proper nouns, prepositions, and plurals were discarded using the natural language processing tool to improve the quality of the database. The words were then stored into corresponding databases in the Combinator according to their part of speech. These databases are ‘the core database’ of the Combinator, as shown in Figure 2.

In order to construct a human-like associative knowledge database, the core database is required to be linked with a semantic net. ConceptNet is used in this project, which is a knowledge base that provides a large semantic net representing general human knowledge and the commonsense relationships between them (Liu & Singh 2004; Speer & Havasi 2012). The knowledge contained in ConceptNet is mined from crowd-sourced resources (for instance, Wikipedia, Wiktionary, and Open Mind Common Sense), expert-created resources (such as, WordNet (Fellbaum 1998) and JMdict (Breen 2004)), and other resources (such as DBPedia (Lehmann et al. 2015)). Besides, ConcepNet allows users to add data and even to create their own versions.

The information elements in the core database are unorganised and unrelated, for example, eight individual elements from the core database are shown in Figure 3. After connection with the ConceptNet, the core knowledge elements are connected to each other as well as linked with information that is correlated but not included in the core database. This associated network formed the Combinator database. For instance, the eight core items of information are linked with each other as well as associated with correlated information via commonsense relationships such as ‘Is A’, ‘Used For’, ‘At Location’, and ‘Related To’, as shown in Figure 4. The associative network example shown in Figure 4 is an epitome of the Combinator database, which demonstrates how information is structured in the Combinator database.

Figure 3. Information elements in the core database.

Figure 4. Information elements in the Combinator database.

3.3 Establishing the Combinator module

As shown in Figure 2, the Combinator module provides an imitation of aspects of human short-term memory while the human mind is processing combinational creativity. It retrieves information from the Combinator database and combines it with the user input to produce combinational ideas. The user input can be a desire or a design keyword, and the output idea is a text combination along with a correlated image combination. The Combinator module can provide a cue, provided by the user input, to retrieve a word or a phrase that is associated with the input from the Combinator database to replace the input for the idea combination. For example, if the user input is ‘chair’, the Combinator module will retrieve an associated word, such as ‘sofa’ or ‘wood’, and combine it with another core information word retrieved from the core database which was crawled from the Internet. This replicates the human mind’s associative power that people often think of other information associated with the one in the mind. For instance, when someone comes up with ‘car’, he or she might also think of ‘window’, ‘seat’, ‘park lot’, and so forth.

The Combinator module can produce a noun–noun combinational phrase, as well as adjective–noun compound phrase and verb–adjective–noun combinational phrase. The use of language or words as stimuli to increase creativity has been previously explored (e.g. Chiu & Shu (2012)). However, producing only text-based combinational ideas can have limitations, as sometimes it is difficult for a user to capture the combinational meaning of a compound phrase. This might be caused by the limitation of the user’s vocabulary or the lack of corresponding images in the user’s long-term memory. Combinational images provide an opportunity to enhance the understanding and interpretation of a generated compound phrase, as the human brain activities are mainly triggered by the input gathered visually (Luis-Ferreira & Jardim-Goncalves 2013). Superimposed and merged images, which are regarded as combinational images in this study, can lead to more creative outcomes (Ward & Kolomyts 2010). Combined images have been identified as effective stimuli for creativity (Ward & Kolomyts 2010).

Images need to be gathered from a source such as the Internet or an image database in order to generate combinational image-based ideas. The open-source crawler used can crawl images according to the existing text-based database and store them in an image database, but creating an image database via the web crawler can take weeks or even months as it is usually enormous. Therefore, the authors have developed a live feed image crawler using Python software to be more time-efficient. The live feed image crawler, which is integrated into the Combinator module, can retrieve corresponding images in real time from the Internet according to the generated combinational phrases.

The Combinator is capable of producing combinational image-based ideas with the assistance of a vision algorithm library. The Combinator provides two main methods for image combination: direct combine and blending. In direct combine, images are cropped first and then merged next to one another. In blending, images become transparent first and then superimposed on one another. For example, Figure 5 shows the original images of an ice cream scoop and a shell, and their direct combination image as well as the blending combination image. Ideas can be derived from the combinational images, such as an ice cream scoop based on a nautilus shell for serving (Millar, Goltan & Cheng 2010). The Combinator implemented the alternative image combination methods to avoid the Einstellung effect that might block the mind of sticking to just one type of image-based idea combination.

Figure 5. An example of image combinations.

Ward & Kolomyts (2010) point out that mentally combined visual-based ideas are essential for developing inventions and exploring discoveries. Therefore, the combinational ideas produced by the Combinator, especially the image-based ideas, are potentially beneficial for designers in idea generation. Users of the Combinator can be further stimulated by the generated combinational ideas, and thereby producing or deriving more ideas.

3.4 Examples of using the Combinator

The Combinator software, in the formulation described here, provides a simply designed user interface, as shown in Figure 6. There is an input box at the top that allows users to input their design keyword or desire. The Combinator provides three option menus respectively for selecting a semantic relation, choosing the number and the part of speech of words used for combination, and selecting an image combination method. The semantic relation option menu can be switched on or off by the switch button next to it, which allows users to choose a relationship from over twenty types, for instance, ‘Related To’, ‘Is A’, ‘Part Of’, ‘Member Of’, ‘Has a’, ‘Used For’, ‘Capable Of’, ‘At Location’, ‘Causes’, and ‘Synonym’. Thereby, the Combinator can retrieve information associated with the input keyword in a selected relation. However, the retrieved word or information might not belong to the same part of speech category as the input word. There is also a ’Random’ option which can choose a relationship randomly from the list. The ‘Select what to combine with’ menu is used for defining the combination output type, for example, to combine the input with ‘one noun’, ‘two nouns’, or ‘verb $+$ adjective $+$ noun’. Users can select the output combinational image style, ‘Direct Combine’, ‘Blending’, and ‘Random’ (Randomly select a method), via an image combination method menu.

Three function buttons, ‘Generate’, ‘Show Image/Change Direction’, and ‘Play’, are located at the bottom of the interface. A text-based combinational idea is produced after clicking the ‘Generate’ button. For example, the combinational idea ‘Origami Handbag’ is generated according to the input ‘Handbag’, as shown in Figure 6 (while the semantic relation function has been switched off). A correlated image-based combinational idea pops out while ‘Show Image/Change Direction’ is clicked. For instance, a combinational image of ‘Origami Handbag’ is generated, as shown in Figure 7. The combinational image produced can be saved on the user’s computer. By clicking the button again, the image combination method or direction is changed to gain more stimulus, for example, blending can transform into direct combine, and left and right merging direct combine can change into up and down merging. The ‘Play’ button yields a slideshow of combinational ideas, including both image-based and text-based ideas, correlated with the input keyword. New information is retrieved from the database for producing a series of new combinational ideas according to the input. This can help users improve divergent thinking and avoid design or cognitive fixation.

Figure 6. The Combinator user interface.

Figure 7. A generated combinational image.

In the current version of the Combinator, the combinational ideas are produced in a random manner to provoke the users’ brain. In other words, ideas are randomly selected from the database to be combined with the input to produce the combinational creative idea. Randomness has been identified as a vital element of creativity (Carruthers 2011), and random stimuli have been indicated to be beneficial in ideation (Howard et al. 2011).

As shown in Figures 6 and 7, a handbag idea, an origami handbag, has been prompted by the Combinator. A number of ideas can be interpreted or derived from the generated compound phrase and the combinational image, for example, a handbag that can be folded similar to origami (Kawamoto 2013). As a result, it is expected that numerous ideas related with a handbag can be generated by the Combinator.

Another example of generating ideas about a ‘violin’ is shown in Figure 8, while the semantic relation option is switched on and ‘Has A’ relation is selected. The Combinator has retrieved a semantically related idea ‘String’, as ‘Violin’ has ‘String’. This is similar to the human while a person has ‘Violin’ in the mind, he or she might think of ‘String’. According to the generated outcome ‘Spider-Silk String’, ideas, such as a violin made using spider silk for string or for influencing the properties of the construction, can be produced or implied (Alesandrini 2016). The simple examples demonstrated in this section have shown how the Combinator works and indicated its capability of assisting users in generating creative ideas.

Figure 8. An example when the semantic relation is switched on.

4 Evaluation

In order to evaluate the usefulness and effectiveness of the Combinator as well as the creativity level of the ideas produced by using the tool, we have conducted a controlled experiment case study of individual designers tackling a novel design challenge. The case study compares participants generating ideas by using the Combinator with participants not using any tools. It also compares participants producing ideas by using the Combinator and participants using Google Image, as both of the tools provide images.

Evaluation of idea generation methods or tools can be grouped into process-based and outcome-based approaches (Shah, Smith & Vargas-Hernandez 2003). A process-based evaluation is not geared to analyse outcomes, but rather used to analyse how and why the ideation method produces the outcome. An outcome-based evaluation is focused on analysing outcomes, which can indicate the characteristics of the idea generation method. In order to understand the Combinator in both output and process aspects, both outcome-based evaluation and process-based evaluation approaches are used in this study.

4.1 Outcome-based evaluation

An outcome-based approach is the most common method for evaluation, for example the study conducted by Istre et al. (2013). The outcome approach is easier than the process approach, especially for engineers, as there are fewer difficulties for experts to evaluate a set of ideas in their domains for a given problem (Shah et al. 2003). In an outcome-based evaluation of creativity, psychometric measurements have been used extensively for decades. A good measure of the outcomes can objectively reflect the idea generation method, as specific metrics can be used to relate the creativity of ideas to the performance of the ideation method.

Researchers have proposed several sets of psychometrics to evaluate the creativity level of outcomes generated by an ideation method, for example, Plucker & Makel (2010) used fluency, flexibility, originality, and elaboration of ideas to measure creativity. Dean et al. (2006) indicated a set of psychometric composed of novelty, workability, relevance, and thoroughness for creativity measurement. Diedrich et al. (2015) only used novelty and usefulness for creativity evaluation. In the study conducted by Chulvi et al. (2012), novelty and utility were applied for creativity assessment through using a questionnaire based on the CPSS (Creative Product Semantic Scale) (Besemer & O’Quin 1989). The taxonomical form of this approach allows to select which items to use in each assessment. Sarkar & Chakrabarti (2011) employed novelty and usefulness to assess creativity of products. In this method, novelty is determined by the FBS (Function–Behaviour–Structure) model and the degree of novelty is assessed by the SAPPhIRE model (Chakrabarti et al. 2005), while usefulness is evaluated by level of importance, rate of popularity of use, frequency of usage, and duration of use. It has been indicated that this method can reflect the intuitive notion of novelty and usefulness of experienced designers.

Shah et al. (2003) proposed quantity, quality, novelty, and variety as the four creativity measures. In this method, quantity is the total number of ideas produced, which is known as fluency in idea generation. It is commonly considered that the more ideas generated, the higher chance of better ideas occurring. Quality, which is commonly described by usefulness (Kudrowitz & Wallace 2013), is the feasibility of an idea and how close the idea comes to meet the design specifications. High-quality ideas generally have higher design success rates. Novelty is the unexpectedness or unusualness of an idea comparing with the others, which implies originality. A novel idea is usually the result of expanding the design space. Variety or flexibility reflects the exploration of the solution space during an ideation process. Generating similar ideas indicates low variety, which shows a limited exploration of ideas in other areas of the solution space. It has been indicated that this approach is focused on measuring the creativity of solving design problems and the effectiveness of an idea generation method, while other approaches, such as Sarkar & Chakrabarti (2011), are focused more on measuring the creativity of products or outcomes. Therefore, Shah et al.’s (2003) psychometric evaluation method was chosen in this study to evaluate the Combinator.

4.2 Process-based evaluation

Compared with an outcome-based approach, a process-based approach is time-consuming and difficult, because observations of the occurrence of creative cognitive processes are required while designers are producing ideas. In addition, there are no generally agreed protocol studies that can be used to collect data for idea generation process evaluation (Shah et al. 2003). Simple cognitive models exist, such as the Geneplore Model (Finke, Ward & Smith 1996), and the Roadmap Theory (Smith 1995), which can be used for analysing data in simple tasks. These models could not be applied to ‘complex’ problems, for example, challenges involving engineering knowledge.

However, a process-based evaluation can demonstrate how an outcome is achieved, for example, the study conducted by Fiedler et al. (2015). Observation is used as a process-based evaluation method in this study, as it can assist in understanding the actual uses of a new technology and detecting potential problems (Yin 2013). Video recording and screen recording are used in the case study to assist the observation. To some extent, the observation result can demonstrate how and why designers produce ideas through using the Combinator.

Table 1. Basic participant information.

4.3 Design challenge and participants

A practical design challenge was introduced to evaluate the Combinator. Waste separation for recycling at homes and offices is necessary. Two or more dustbins are often used to achieve the goal, but this is often space consuming and messy. The design challenge was to design a new solution that can efficiently use space as well as provide waste disposal and recycling attributes. Customer needs, such as waste separation, space saving, easy of use, no unpleasant odours, and stylish appearance, were provided, which were regarded as design specifications in this design challenge.

Thirty-six individuals, who were familiar with ‘dustbins’, participated in this case study and the following interview. The participants were highly interested and intrinsically motivated to participate in this idea generation case study voluntarily. They have signed up with standard case-study protocols concerning use of data and giving permission for the use of HD video recording. They were also rewarded with two pieces of high-quality stationery after completing the challenge and the interview. The basic information of the participants is shown in Table 1. Six of the participants are considered as experienced designers having over three years of design experience. The other thirty participants are regarded as novice designers, as none of them has sufficient experience in either engineering design or industrial design. Twelve of the participants conducted the study by using the Combinator, another twelve participants were not using any tools, and the remaining twelve used Google Image. The participants were regarded as the Combinator participants, the non-tool participants, and the Google Image participants, respectively. In order to have a fair competition, the division of participants was based on the participants’ experience and background, and thereby constituted three categories possessing similar capabilities.

4.4 Evaluation of the Combinator

The thirty-six participants conducted the challenge separately, as a person’s performance may be influenced by the action and ideas of others while producing ideas in groups (Perttula & Sipilä 2007) in a quiet room without any interruptions. Each of them was provided with A3 paper and a pen for sketching and writing ideas. The participants were given the same amount of time to generate as many ideas as possible. The Combinator participants generated ideas under the assistance of using the Combinator and was trained to use the tool before starting the challenge. The non-tool participants came up with ideas based on their intuition and experience, while the Google Image participants produced ideas by using Google Image. During the task, all the participants were observed by a recorder and recorded with high-definition cameras. The observations were conducted silently in order to minimise the impacts on the participants. Interviews were conducted after each participant had accomplished the design challenge. The generated ideas from each participant were collected and mixed together before the evaluation in order to eliminate bias.

Sarkar & Chakrabarti (2011) indicated that identifying creativity of a product is challenging, as one cannot be knowledgeable enough about all products. Experienced designers are often used to evaluate creativity of ideas, concepts, and solutions in design companies. It was also pointed out that the experienced designers who are selected to judge creativity should come from similar domains. Therefore, two experienced design engineers having over three years’ experience in design engineering were selected to evaluate the ideas respectively. The ideas were evaluated under the same guidance and the same inter-rater agreement of scoring 8 to 10 for excellent, 5 to 8 for good, 3 to 5 for fair, 1 to 3 for poor. The final scores are the means of the scores graded by the two raters.

The four metrics, quantity, novelty, quality, and variety, were measured according to Shah et al. (2003). Quantity was measured by counting the total number of ideas generated by an individual. Variety was applied to the entire group of ideas produced by an individual, which was measured by counting the total number of idea categories. The ideas generated were grouped based on the different physical principles used to satisfy waste separation which made the ideas very different from one another. For instance, using a chest of drawers to store waste and using barrels to store waste for satisfying waste separation were grouped into two categories, and thereby indicate two idea varieties.

Novelty of an idea was assessed by scoring each key function 1 to 10 from ‘poor’ novelty to ‘excellent’ novelty while comparing with the existing conventional dustbins on the market. An image including a collection of these conventional bins for waste separation (a swing bin for general waste, a caddy bin for food waste, and two rectangular recycle bin for paper, plastic, and glass) was provided to assist comparison. These bins are commonly used in people’s daily lives, and were considered as not novel in this study. Waste separation and space saving were considered as the two key functions of a new dustbin idea. These two functions were applied with the same weight 0.5 (total weight 1), as they are equally significant to a new dustbin design. The coefficients of the weights of novelty were decided through consulting a professional design expert who has over 10 years’ practical experience in consumer product design. A higher novelty score would be given to an idea’s ‘waste separation’ or ‘space saving’ function, if the function was more unusual or unexpected in comparison with non-novel ideas. In other words, ideas with higher novelty scores have less overlapping elements with non-novel ideas. The overall novelty of each idea was calculated by using Eq. (1) (adapted from Shah et al. (2003)). $M_{1}$ is the overall novelty score for an idea with $n$ functions. In (1), $f_{i}$ is the weight assigned to the function $i$ , while $S_{i}$ is the score of function  $i$ . The novelty score of an individual participant was the mean novelty score of all the ideas generated.

(1) $$\begin{eqnarray}M_{1}=\mathop{\sum }_{i=1}^{n}f_{i}S_{i}.\end{eqnarray}$$

Quality of an idea was evaluated by scoring each key attribute 1 to 10 from ‘poor’ quality to ‘excellent’ quality considering the idea’s feasibility and to what extent each attribute meets the design specifications. A total weight of 1 was assigned to 6 key attributes according to their importance to the idea as follows, feasibility (0.25), waste separation (0.25), save space (0.25), easy to use (0.1), no odours (0.1), and stylish appearance (0.05). The coefficients of the weights of quality were decided through consulting a professional design expert considering that feasibility, waste separation, and save space were more important than the other design specifications for an initial idea. A higher quality score would be given to an idea, if the idea’s attributes could meet the design specifications in better degrees respectively. In other words, an idea with a higher quality score has a higher feasibility and usability. The overall quality of each idea was computed with Eq. (2) which was adapted from Shah et al. (2003). $M_{2}$ is the overall quality score for an idea including $m$ attributes. $S_{\!j}$ is the score of attribute  $j$ , while weights ( $f_{\!j}$ ) were assigned to each key attribute. The quality score of an individual participant was the mean quality score of all the ideas generated.

(2) $$\begin{eqnarray}M_{2}=\mathop{\sum }_{j=1}^{m}f_{\!j}S_{\!j}.\end{eqnarray}$$

The individual level mean scores of quantity, novelty, quality, and variety of each participant categories were calculated respectively for comparison. It is more effective to compare the scores at the individual level than the totals scores of a category, due to the different amount of ideas produced by each participant.

However, in this evaluation, new functions or attributes, other than the ones stated for evaluation, which might benefit the product, were not considered. This is due to only key attributes or functions are used for evaluation according to Shah et al.’s (2003) psychometric method. Moreover, in initial design phases, it is significant to assess whether a product’s key attributes can satisfy the design requirements. In this study, subjectivity and potential biases might originate during the evaluation, due to issues such as subjective judgements caused by an evaluator’s understanding and perception of the attributes, inconsistent evaluation standards caused by insufficient time as well as inconsecutive evaluation processes, and misunderstanding of ideas caused by vague idea demonstrations and descriptions. Therefore, two experts were introduced to evaluate the Combinator, in order to minimise evaluation biases.

4.5 Results of evaluation

4.5.1 Kappa test

A Kappa test was conducted in order to measure the agreements between the two raters. The test result showed that the kappa coefficients of quantity, novelty, quality, and variety were 1, 0.57, 0.72, and 1, respectively. It indicated that the two raters had an almost perfect agreement on quantity and variety, a substantial agreement on quality, and a moderate agreement on novelty. This indicated that it is valid and significant to use the means of the scores graded by the two raters as the final scores.

4.5.2 Statistical analysis

Based on the psychometric evaluation method and the equations illustrated above, the mean scores of quantity, quality, and novelty at the individual level of each category were calculated and presented in Figure 9. The Combinator participants generated 4.42 ideas at the individual level, while the non-tool participants generated 2.17 ideas and the Google Image participants produced 1.75 ideas at the individual level. The mean novelty score of the Combinator participants was 6.78 which is 0.43 and 0.82 higher than that of the non-tool participants (6.35) and the Google Image participants (5.96) at the individual level, respectively. The non-tool participants and the Google Image participants achieved 5.87 and 5.83 respectively on quality at the individual level, while the Combinator participants scored a 6.67. 3.42 idea categories were demonstrated at the individual level among the Combinator participants, whilst 1.67 varieties were shown by the non-tool participants and 1.50 varieties were presented by the Google Image participants at the individual level.

Figure 9. Psychometric evaluation results.

The standard deviation was introduced to quantify the dispersion of the data values. The standard deviation of the number of ideas (quantity) generated by each participant from the Combinator participants was 1.97, while the standard deviations of the non-tool participants and the Google Image participants were 1.34 and 0.97, respectively. The standard deviation of novelty scores and quality scores at the individual level of the Combinator participants, the non-tool participants, and the Google Image participants were 0.74, 0.71, 0.96, and 0.52, 0.92, 0.76, respectively. The individual level variety scores of the Combinator participants, the non-tool participants, and the Google Image participants were 1.38, 0.67, and 0.98, respectively.

Although the means are different, this might occur by chance. Therefore, statistical analysis is required to determine whether there are statistically significant differences between the means. The statistical analysis is conducted by using IBM SPSS Statistics software and the significance levels of the statistical tests are set as 5% ( $\unicode[STIX]{x1D6FC}=0.05$ ) as a convention. A Shapiro–Wilk test is conducted to analyse whether the data of each metric of the three participant groups are normally distributed. In the test result, if a $p$ -value is greater than 0.05, it indicates that the corresponding data obey normal distribution. As shown in Table 2, the Shapiro–Wilk test result shows that all the data of the Combinator participants are normally distributed, while only the novelty and quality values of the non-tool participants and the Google Image participants are normally distributed, respectively.

Table 2. Shapiro–Wilk test result of data normal distribution.

Table 3. Independent sample T-test result of ‘Novelty’ and ‘Quality’.

An independent sample T-test is conducted to determine whether there are significant differences between the means of novelty and quality, which requires the assumption of normal distributions. During the T-test, a Levene’s test is conducted to assess homogeneity of variances for selecting the proper $p$ -values produced by SPSS. In the test result, if a $p$ -value is less than or equal to 0.05, then there is a significant difference between the means of two conditions. The results of the T-test are shown in Table 3. Comparing the Combinator participants with the non-tool participants, the $p$ -value of the mean quality value is less than 0.05, while the $p$ -value of the mean novelty score is 0.162 which is greater than 0.05. Therefore, there is a significant difference between the mean quality scores and a non-significant difference between the means of the novelty scores. With regards to the Combinator participants and the Google Image participants, the $p$ -values of the two metrics are all less than 0.05. This determines that there are significant differences between the means of novelty and quality, respectively.

A Mann–Whitney U test is conducted to determine whether there are significant differences between the means of quantity and variety. The Mann–Whitney U test is a nonparametric test, which does not require the assumption of normal distributions, used for assessing significant differences. Similar to the independent T-test, there is a significant difference between the means of two conditions, if the $p$ -value is less than or equal to 0.05. As shown in Table 4, all the $p$ -values are less than 0.05. This indicates that there are significant differences between the means of quantity and variety respectively, while comparing the Combinator participants with the non-tool participants and with the Google Image participants.

Table 4. Mann–Whitney U test result of ‘Quantity’ and ‘Variety’.

The effect sizes (Cohen’s d) are calculated to define the differences between the means to support the statistical analysis. The independent T-test has indicated a non-significant difference between the mean novelty scores of the Combinator participants and the non-tool participants. As shown in Table 5, comparing the Combinator participants and the non-tool participants, there is a medium effect of the non-significant difference between the mean novelty scores. However, there are large effect sizes of significant differences between the mean scores of the other three metrics. In terms of the Combinator participants and the Google Image participants, there are large differences between the means of all four metrics.

Table 5. Effect sizes (Cohen’s d) between the metric of different participants.

We have also calculated the robust (bootstrap) confidence intervals, as presented in Table 6, by using 95% confidence interval of means and 1000 samples. It provides robust confidence intervals for estimates based on one thousand samples. The result shows that 95% of the Combinator participants could score 3.21 to 5.50 in quantity, 6.34 to 7.15 in novelty, 6.39 to 6.95 in quality, and 2.58 to 4.12 in variety. The performances of the Combinator participants are better than the other two types of participants based on an estimated 1000 samples data size, according to the calculation result.

Table 6. Robust confidence intervals by using 95% confidence interval of means.

According to the statistical analysis above, in terms of comparing the twelve participants using the Combinator with the twelve participants without using tools concerned in the case study, there are significant increases in quantity, quality, and variety at individual levels. However, there is a non-significant improvement in novelty. Comparing with the twelve participants using Google Image at the individual level, there are large improvements in all four aspects. This indicates that, concerning the conducted design challenge, the Combinator had improved the users’ fluency in idea generation as well as enhanced the originality, usefulness, and flexibility of the ideas produced in comparison with the non-Combinator users.

4.5.3 High-novelty and high-quality ideas analysis

In addition, the number of high-novelty and high-quality ideas are also significant criteria, as generally only high-novelty and high-quality ideas will be accepted and developed into final products. In this case study, an idea is considered as a high-novelty idea or a high-quality idea while its novelty score or quality score is greater or equal to 7, respectively. As shown in Figure 10, the twelve Combinator participants generated 28 high-novelty ideas out of 53 ideas in total, while the twelve non-tool participants only produced 9 out of 26 and the twelve Google Image participants generated 6 out of 21. In terms of high-quality ideas, the twelve non-tool participants and the twelve Google Image participants came up with 3 and 1, respectively, while the twelve Combinator participants generated 19 in total. Thus, 53% of the ideas generated by the Combinator participants are high-novelty ideas, while only 35% and 29% of the ideas generated by the non-tool participants and the Google Image participants are high in novelty, respectively. The Combinator participants generated 36% high-quality ideas, which is about three times of that of the non-tool participants and seven times of that of the Google Image participants.

Figure 10. High-novelty and high-quality ideas.

4.6 Observations and interviews

Observations and video recordings were introduced as a process-based evaluation method. Although this method cannot provide very convincing results, to some extent, it could be used to understand how ideas were produced by different groups of participants during the idea generation sessions. According to observations and video recordings, the recorder would need to record the time a participant spends on sketching and writing ideas, as well as the time a participant spends on thinking (not producing ideas). The recorder would also need to record the reactions of the Combinator participants and the Google Image participants, such as starting to produce ideas, while corresponding types of images are shown to the participants. The results have shown that the participants who were using the Combinator spent less time than the other participants on the thinking process. This could be that non-Combinator users had to retrieve relevant knowledge to produce a creative idea, which could be challenging as well as time-consuming, especially for novice designers. In addition, the Combinator users could always come up with new ideas after being stimulated by the ideas produced by the tool. This implies that ideas or stimuli generated by the Combinator could be used for further idea derivation. According to Howard et al. (2011), a stimulus does not directly inspire new ideas, but divert designers into new design spaces to enable new ideas. Thus, the process-based evaluation has revealed that it was the ideas generated by the Combinator that assisted users in generating creative ideas. This is how the Combinator functioned during the idea generation challenge and the reason why the Combinator users generated more creative ideas than the others.

Interviews were conducted after each participant had accomplished the design challenge. The participants were asked to evaluate their ideas, user experience, and creativity levels during the study, which was called participant evaluation. Scatter charts were employed to demonstrate the individual evaluation results. Please note that the lines in the charts do not represent data variations or trends, the lines are only used for better presentations. In the charts, the vertical axis represents the evaluation scores, while the horizontal axis represents the participant ID from 1 to 12 of the Combinator participants, the Google Image participants, and the non-tool participants, respectively.

Figure 11. Participants evaluation of outcomes: the Combinator VS. Google Image.

Figure 12. Participants evaluation of user experience: the Combinator VS. Google Image.

The Combinator users as well as the Google Image users were asked to score the ideas produced by themselves, from 1 to 10 (from ‘poor’ to ‘good’) in terms of quality, novelty, and variety. As shown in Figure 11, in general, the ideas produced by the using Combinator have higher scores in all three aspects comparing with the ideas produced by using the Google Image. This indicates that the users were satisfied with the outcomes of the Combinator. These participants were also asked to evaluate their user experience of the tools in four aspects: usefulness, easiness, comfort, and enjoyment. According to the user experience evaluation, as shown in Figure 12, the Combinator had provided a superior user experience and been considered as a useful, easy, comfortable, and enjoyable tool. In terms of Google Image, it had been shown as an easy and comfortable tool, but unenjoyable and less useful. All the participants were asked to grade themselves from 1 to 10 to describe how creative they feel during the idea generation sessions. As shown in Figure 13, overall, the Combinator participants had graded themselves with higher creativity level scores comparing with the others. This implied that the Combinator had positively influenced the creative thinking of the participants during idea generation.

Figure 13. Participants evaluation of creativity level: the Combinator VS. Google Image VS. Non-Tool.

All the twelve Combinator participants provided highly positive feedback considering the Combinator as a very useful tool for idea generation. However, some participants using the Combinator pointed out that the qualities of several images produced from the Combinator were poor, which interfered with their thinking. Several non-design background participants indicated that the Combinator generated many high-quality and useful ideas, but it was difficult for them to translate these ideas into creative outcomes. Most of the Google Image users considered Google Image as a useless tool, as the images provided by the tool were monotonous. In terms of the non-tool participants, some thought that the design task had a certain degree of difficulty, and thus it was challenging for them to come up with a number of creative ideas. Most of the non-tool participants acknowledged that they need some stimuli, especially visual stimuli, to help them in idea generation.

4.7 Discussions

Several examples of the ideas produced by using the Combinator have been selected and sketched, as shown in Figures 14–17. Figure 14 shows a combinational idea for a ‘Slide Bin’. The originator of this ideas noted that the Slide Bin was inspired from the combinational idea ‘slide – bin’ generated by the Combinator. Consumers can spin the slide to choose the type of waste to dispose. This creative design provides waste separation function as well as the enjoyment of allowing waste to go down the slide to a selected destination. The ‘Tangram Bin’, as shown in Figure 15 was generated based on the combinational output ‘tangram – bin’. The space-efficient tangram design allows the bins to connect to one another freely to form a variety of shapes, which is similar to tangram puzzles. The design enables customers to custom their bins’ waste classification and layout. A ‘Flower Bin’ idea and a ‘Stair Bin’ idea are shown in Figures 16 and 17, respectively. These two ideas were also generated under the assistance of the Combinator. The examples indicate that the Combinator is capable of assisting designers in producing creative ideas that are quality, novel, and useful.

Figure 14. The slide bin.

Figure 15. The tangram bin.

Figure 16. The flower bin.

Figure 17. The stair bin.

As illustrated in the evaluation sections, comparing with the ideas generated by the designers using intuition and experience as well as Google Image, the ideas produced by the designers using the Combinator are better in quantity, novelty, quality, and variety. Therefore, the Combinator is considered to be beneficial for designers in idea generation, albeit based on a limited data sample. The new tool is capable of assisting designers in increasing better idea occurrence, expanding the design space, enhancing the design success rate, and improving design space exploration. The improvement on the four metrics has reflected the effectiveness and the creativity of using the Combinator on solving design challenges. The case study has demonstrated that the Combinator could enhance the users’ fluency in idea generation as well as increase the originality, usefulness, and flexibility of the ideas produced. Additionally, the test results of the participants using Google Image are slightly poorer than the results of the participants using knowledge and intuition. The monotonous images provided by Google Image might had led some participants into design fixation.

According to the statistical analysis, there were large improvements on quantity, quality, as well as variety, while comparing the ideas produced by the Combinator participants with the ideas produced by the non-tool participants as well as the Google Image participants. There was a significant increase in novelty while comparing the Combinator participants with the Google Image participants, but a non-significant increase in comparison with the non-tool participants. The statistical analysis indicates that the ideas produced by the Combinator had significantly improved the design space exploration, better ideas occurrence, and design success rate. Westerlund (2009) illustrated that the exploration of the design space is conducted through exploring possible solutions. Thereby, we can assume the ideas produced by the Combinator had significantly increased the number of possible solutions which supported the users in the design space exploration. The ideas generated by the Combinator had stimulated its users to produce more ideas. A greater number of ideas generated indicates a higher number of better ideas. The better ideas produced could lead to a higher design success rate. However, the tool only slightly expanded the design space. It has been revealed that the design space can be expanded by introducing new design variables and stimuli (Gero & Kumar 1993; Howard et al. 2011). Considering the conducted design challenge, the design specification had reached a high degree, therefore, there was a low possibility of adding new design variables. The outcomes of the Combinator were considered as new design stimuli, but some designers might have difficulties in recognising the stimuli or transforming them into novel ideas.

As shown in Figure 10, the number and the proportion of high-novelty ideas and high-quality ideas generated by using the Combinator are greater than the ones produced without using any tools and the ones produced by using Google Image, respectively. Thus, in this case study, a greater number of ideas generated by using the tool can be implemented into final designs.

The observations indicated that the Combinator users generated more outcomes than the others with the help of the ideas produced by the tool. Through interviewing the participants, the designers from the Combinator participants felt more creative than the others during the ideation, which indicated a positive unconscious influence of the Combinator. The users of the Combinator were highly satisfied with its user experience and outputs. However, a few users suggested that the output image quality can be improved.

The case study shows that all the Combinator users provided highly positive feedback, considering the Combinator as a useful, easy, comfortable, and enjoyable tool which can produce quality, novel, and various outcomes. The process-based evaluation reveals that the Combinator users were inspired by the ideas generated by the tool, and thereby generating or deriving creative ideas. Albeit based on a limited data sample, the outcome-based evaluation results suggest that the Combinator can help designers generate creative ideas which are outstanding in quantity, quality and variety.

In long idea generation sessions, the Combinator is considered to have significant advantages, as it can continuously generate prompts, which can be regarded as stimuli, to assist users in ideation. According to Howard et al. (2011), several industrial case studies showed that the idea generation rate declines after 30 min, and idea quality decreases dramatically after 20 min during brainstorming. It was also indicated that the use of stimuli during brainstorming could maintain the idea generation rate and quality during brainstorming. Therefore, it is highly possible that the longer the hours of idea generation, the better performance of using the Combinator.

The effectiveness of the Combinator can be affected by multiple factors, such as users’ background, users’ experience, users’ age, and the design problem. In terms of the conducted case study, it is the design challenge that can be deemed as the most critical factor in terms of maintaining high creative levels, as the design challenge had reached a high degree of design specification. At the moment, the tool is under testing before the initial public release, but copies of the tool are available on request from the authors.

5 Conclusions

Combinational creativity is the easiest and the most commonly used approach for humans to achieve creativity, but has been elusive in computational tools to date. The tool is an imitation of the aspects of the human brain in generating combinational creative ideas. It produces combinational ideas in texts as well as in images through combining unrelated or indirectly related familiar ideas automatically similar to humans. This study has utilised up-to-date information retrieval technology, natural language processing tool, and existing knowledge base for creating the Combinator, a computer software tool, based on combinational creativity.

A case study was conducted to evaluate the Combinator. Through observations and video recordings, it was noticed that the participants who were using the Combinator spent less time than the other participants on the thinking process. All the Combinator users provided positive feedback in terms of user experience and outputs. The study results have indicated that the Combinator can be useful and effective in assisting novice designers as well as experienced designers in idea generation. Compared with non-Combinator users, the Combinator has improved the users’ fluency in idea generation and enhanced the originality, usefulness, as well as flexibility of the ideas produced, for the participants concerned. The improvement of the four metrics has indicated the high effectiveness of using the Combinator in idea generation. The limited case study undertaken indicates that the Combinator is considered to be beneficial in expanding the design space, increasing better ideas occurrence, improving design space exploration, and enhancing the design success rate. Nevertheless, the degree of improvement through using the Combinator is based on different design challenges and different tool users. The relatively high proportion of high-novelty ideas and high-quality ideas generated by using the tool indicates that a greater number of ideas produce by using the Combinator can be developed into final designs comparing with without using the tool. The simulation algorithm used for developing the Combinator can be considered as a new effective approach for developing design tools, especially idea generation tools. Further research is planned to improve the efficiency and effectiveness of the Combinator, as well as to enhance the Combinator algorithm in imitating aspects of the human brain in generating combinational ideas.