Is virtual reality so user-friendly for non-designers in early design activities? Comparing skills needed to traditional sketching versus virtual reality sketching

Abstract

Virtual reality (VR) sketching has many advantages for product design and tends to be more and more used among designers and non-designers (end-users). Nevertheless, few studies have focused on the skills needed to use VR sketching for non-designers especially VR novices in VR software. This study focuses on identifying the cognitive impact of VR sketching compared to traditional sketching on VR expert and VR novice in an experimental setting. Thirty-one participants composed of VR experts (N = 15) and VR novices (N = 16) completed a mental rotation test and then performed one traditional paper and pencil sketching task and two VR sketching tasks. We also measured the participants’ movements when using the VR sketching. Results show that VR experts perform better than VR novices in VR sketching because training is an essential element for the quality of traditional and VR sketching. Nevertheless, VR novices with previous training in traditional drawing and/or high mental rotation skills will be able to produce good-quality sketches. In addition, the results show that users moving more in the immersive environment performed better quality sketches if the drawing requires more complex shapes. Our results suggest that VR sketching can be complex to use for a part of the population that may be end-users, especially for those with little experience in traditional and VR sketching and with poor visuospatial abilities. We, therefore, advise to check the non-designers’ prior skills, otherwise, it will be necessary to train these users in VR sketching.

1. Introduction

Consumers/users seem to be increasingly involved in societal consumer issues (Sikhwal & Childs 2018) especially when they are integrated into the design process (Bendapudi & Leone 2003). Consumers/users also appear to want individualized products (Koren et al. 2015; Sikhwal & Childs 2019). To meet this need for individualization, companies are using collaborative design approaches that integrate the user into the design process (Pallot 2011). Beyond user-centered design (ISO 924-210: 2019) or the co-design, Do It Together approach proposes to make designer and non-designer (i.e., end-user) collaborate toward a common, unique and individual goal (Hirscher, Niinimäki & Joyner Armstrong 2018). According to Hirscher et al. (2018) and Dupont et al. (2023), Do It Together is an alternative design strategy to organize the design process toward a more local consumption mode and closer to the end-users who become active consumers through their own idea. According to Koren et al. (2015), this collaboration could result in the creation of openness platforms that users and companies can adapt, designing individual products that fit the user’s need. In this social manufacturing approach, the non-designer becomes a value creator (Hirscher et al. 2018) and should be empowered to perform this function. This means that the non-designer who is an end-user must use and manage design tools, known to designers, without being trained to use them. These tools can be in several forms. For instance, designers are accustomed to using sketching which is an effective method of expressing ideas and a communication medium, essential for the success of the ideation phase (Dorta 2004; Goldschmidt 2014). Sketching is a sub-discipline of drawing and can be defined as a simplified version of this one. In our study, we use the term “sketching” but the underlying cognitive processes studied in this article could also apply to other forms of drawing.

Usually, in the drawing field, the use of a 2D medium is typically used to present three-dimensional (3D) elements. The 2D sketching makes it difficult to design and interact in 3D space and prevents the user from mastering spatial proportions (Dorta, Kinayoglu & Hoffmann 2016). One might therefore think that the use of a 3D medium (such as virtual reality (VR)) would be beneficial compared to the use of a 2D medium (such as paper/pencil). However, in the study of Fleury et al. (2022), the authors show that users will systematically use paper/pencil sketching before making their VR sketches. Participants had the option of skipping this step of paper/pencil sketching. In addition, Kim et al. (2020) show on a VR software developed to support gardener apprentices in designing gardens that the design quality (in terms of proportion and composition aspects) was improved when it was carried out after the traditional paper/pencil sketching. If users need an intermediate step (i.e., paper/pencil sketching), it is probably related to the fact that they are not comfortable with the 3D tool used or cannot project themselves directly into a 3D environment to communicate. It would therefore be interesting to better understand the transfer of skills between 2D and 3D sketching.

Especially in a Do-It-Together design process, digitalized products must be as user-friendly as possible because the complexity of the interface is not only a limit to the use of the product but also to the creative thinking of the user. For example, this is the case with digital tools that force the user to make design choices too early in the process (Dorta et al. 2016). Sketching allows for designers to quickly define problems, explore ideas, mobilize knowledge (Brun, Masson & Weil 2016) and develop new solutions using paper, digital media, or both and foster feedback loops (Goldschmidt 2014) not insignificant in a collaborative project. It is important that sketches are produced in large quantities, quickly, easily discarded or modified and usable for others (Buxton 2007).

The traditional paper and pencil sketching is now being digitized and supports more effective distributed international collaboration (Pallot 2011). The digitalized tool of traditional sketching can take the appearance of VR sketching, which has received great interest in recent years. VR sketching allows collaborative drawing and has other advantages widely referenced in the scientific literature (e.g., Schnabel 2011; Milovanovic et al. 2017; Feeman, Wright & Salmon 2018; Yang & Lee 2020). The VR activities create a sense of presence that leads to better communicate the overall intentions of the designer (Schnabel 2011). Yang & Lee (2020) have shown that VR sketching boosts the creativity of designers in the ideation phase, because these tools allow the extension of the design solution space, improve the transformation of ideas and encourage a holistic approach to design for concept generation. VR sketching also allows sketches to be viewed from other perspectives because users can move around the three-dimensional sketch (Milovanovic et al. 2017). VR sketching seems superior to traditional sketching for creative sketching because it leads to a satisfying level of efficiency, effectiveness, ease of use and enjoyment (Van Goethem et al. 2020). Moreover, with the advent of the Do-It-Together, it is crucial to have easy-to-use communication tools between designers and non-designers in order to exploit their ideas. According to Pallot et al. (2023), the main drawback to the Do-It-Together process is the lack of appropriate customer skills to adequately contribute to design and manufacturing activities. Indeed, the ability to create visual images using freehand interactions remains a fundamental skill for designers (Booth et al. 2016) and also for non-designers that is critically important to the proper understanding of ideas (Dorta 2004). In addition, ideas communicated with high-quality sketches are much more likely to be perceived as creative compared with the same ideas shown with low-quality sketches (Kudrowitz et al. 2012). In fact, VR sketching provides exciting alternatives for creating and expressing new design ideas and communicating visually. However, VR sketching is questionable, would non-designers be able to use it? Improper use of VR sketching could harm the whole process of Do-It-Together.

Some studies have already highlighted collaborative case studies (e.g., de Klerk et al. 2019; Safin 2020; Fleury et al. 2022) involving lay people in a design process via a VR sketching tool. However, to our knowledge, few studies seem to have focused on the skills needed to use VR sketching compared to traditional sketching, especially with non-designers. This suggests that we do not know if VR sketching is suitable for non-designers which is a crucial tool for collaboration between designers and non-designers in a Do It Together platform.

2. Skills needed for traditional sketching

Drawing/sketching is an active, creative and self-directed process leading to a “different way of seeing” (Edwards 1997). Sketching is an integral part of the engineering curriculum for conceptual understanding, communication and design (Merzdorf et al. 2021). Designers need to have the ability to produce quality sketches that can be used by others (Goldschmidt 2014; Booth et al. 2016). It means having drawing skills in two dimensions and three dimensions (e.g., for the sketches of blueprint, engineering drawing and CAD). That is why, we were interested in the skills needed for traditional sketching, then VR sketching.

A significant number of studies suggest that traditional drawing experts have acquired many skills such as better attention span, manipulation skills and better spatial skills (e.g., Orde 1997; Alias, Gray & Black 2002; McManus et al. 2010; Perdreau & Cavanagh 2015; Contreras et al. 2018; Benear et al. 2019; Chamberlain et al. 2021; Park, Wiliams & Chamberlain 2021). Drawing/sketching involves a wide range of major cognitive functions such as:

Spatial skills (e.g., mental rotation, spatial visualization, spatial orientation, spatial memorization, spatial exploration) to analyze, understand and visualize space in two and three dimensions (Linn & Petersen 1985; Tartre 1990; Samsudin, Rafi & Hanif 2011). This article focuses on visuospatial abilities. Alias et al. (2002) were interested in the link between the visuospatial abilities of engineering and architecture students and their sketches. The authors found that visuospatial abilities are directly linked to the tendency to use sketching/drawing and indirectly to students’ view of the professional role of sketching/drawing.

Memory with many subsystems (sensory memory, short-term memory, long-term memory – for example, Atkinson & Shiffrin 1968). For instance, McManus et al. (2010) and Perdreau & Cavanagh (2015) showed that traditional drawing experts have a better visual memory than traditional drawing novices. This would be related to the ability to copy angles and simple proportions (McManus et al. 2010). Spatial abilities and spatial memory are interdependent skills.

Sketching skills take a long time to develop (Booth et al. 2016). For example, perspective drawing is not intuitive. According to Booth et al. (2016), mastery of sketching skills requires training and practice. In a drawing task, when participants reproduce a simple or complex shape, they have to plan the sequence of elements to be drawn and they have to consider the spatial relationships between them (La Femina et al. 2009). From this point of view, drawing can be considered to be a particular type of construction task. This involves developing a strategy and learning how to draw (La Femina et al. 2009).

The use of strategy was evidenced by a visual exploration of specific drawings by trained draftsmen. Park et al. (2021) were interested in the eye movements of artists and non-artists while making drawings representative of photographic stimuli. The authors showed that it was possible to discriminate between the two groups based on their global and local ocular saccades when looking at the target stimulus during drawing. The results showed that these differences in eye movements are not specifically related to figurative drawing ability and may be a feature of artistic ability more generally. This would imply that participants with drawing experience would have greater artistic ability than those without drawing experience. Thus, expertise is generally acquired as a result of deliberate practice (from time spent drawing, to using drawing techniques) and a flexible approach to learning strategies (Chamberlain et al. 2015). Conversely, McManus et al. (2010) show that drawing skills were not related to socio-demographic characteristics. However, the authors showed that the subjective assessment of drawing skills was representative of the actual skills of the drawers. Students who perceived themselves as good drawers drew better than those who perceived themselves as poor drawers.

To sum up, traditional sketching requires several skills mainly including spatial abilities and training.

3. Skills needed for VR sketching

In the previous lines, we mentioned some key advantages of VR sketching, which has also some disadvantages. According to Wiese et al. (2010) and Arora et al. (2017), VR sketching is less accurate than traditional sketching. It seems more difficult to draw a 3D sketch than a 2D sketch. This is due to the absence of a physical surface (Arora et al. 2017) and the “depth perception errors” (Cave & Kosslyn 1993; Tramper & Gielen 2011; Arora et al. 2017), that is, a lack of spatial representation. Compared to traditional sketching which is in 2D, VR sketching which is in 3D requires higher demands on the user’s perception, motor and visuospatial abilities (Barrera Machuca, Stuerzlinger & Asente 2019). In addition, VR sketching requires movements and spatial inspection (Barrera Machuca et al. 2019; Yang & Lee 2020). Yang & Lee (2020) highlight that spatial inspection is a behavioral factor for successful VR sketching. Barrera Machuca et al. (2019) explain that users need to use different views to plan their next hand movement. All of these arguments seem to converge toward the importance of taking into account the spatial environment and visuospatial abilities (e.g., La Femina et al. 2009; Branoff & Dobelis 2013; Barrera Machuca et al. 2019; Obeid & Demirkan 2020). Nevertheless, visuospatial abilities could be improved with training (Dünser et al. 2006; Samsudin et al. 2011). Wiese et al. (2010) showed that the quality of VR sketches improves over time. More specifically, prolonged use of virtual learning environment could improve mental rotation skills (Farzeeha et al. 2017). In contrast, Bolier et al. (2018) found that the quality of VR drawings made by children improves over time but not their visuospatial abilities.

4. Research question and hypotheses

Thus, currently, there does not seem to be a consensus on the impact of visuospatial abilities and training on the use of VR sketching. Only a few studies have investigated the use of VR sketching especially among non-designers. The aim of this study is to better understand the skills needed for traditional sketching and VR sketching for non-designers. To do this, we formulated five hypotheses:

• • First, according to La Femina et al. (2009) McManus et al. (2010) and Chamberlain et al. (2015), training seems to be an essential element for the quality of traditional and VR sketching. We hypothesize that users with high traditional drawing skills (e.g., who received a traditional drawing training) will produce higher quality traditional and VR Sketches than those with low drawing skills.

• • Second, with the same approach, according to Wiese et al. (2010) and Bolier et al. (2018), we hypothesize that VR experts (e.g., who are used to VR sketching) will produce higher quality traditional and VR sketches than VR novices.

• • Third, according to Alias et al. (2002), La Femina et al. (2009) and Barrera Machuca et al. (2019), we hypothesize that the higher the visuospatial abilities (e.g., mental rotation) of the users, the higher the quality of traditional and VR sketching.

• • Fourth, according to Barrera Machuca et al. (2019) and Yang & Lee (2020), movement and spatial inspection are crucial to successful VR sketching. Hence, we hypothesize that the users who move (i.e., users move their body or/and their head in the 3D environment) the most are those who will have a better quality of VR sketching.

• • Fifth, the perceived usability of the software used in this study (Time2Sketch) can have an impact on the quality of the VR sketches. If the software does not have a good perceived usability, then users will have VR sketches of lower quality.

5. Methods

5.1. Participants

Thirty-one participants, 15 females and 16 males aged 18–62 years (means = 34.03 ± 12.75) participated in this study. All the participants were native French speakers and signed an informed consent form. The data collected about participants was anonymous. This study was in line with the ethical recommendations of the Declaration of Helsinki. Participants did not receive financial compensation. The VR experts were recruited from a center of research or a VR compagnies specialized in VR and the VR novices were recruited through a call for studies. The experts work in VR and the novices do not work in this field.

5.2. Materials and measurements

5.2.1. Time2Sketch software

Time2Sketch is an immersive sketching software used in the experiment to allow the user to draw freehand lines in VR. Users can change color, brush size, erase the lines, undo the last action, resize the sketch, teleport in the environment and position a symmetry axis. The VR headsets used was Oculus Quest.

5.2.2. Measure of position and movement

The headset records the following participants’ movement while using the equipment: position in the scene on an x, y and z-axis which allows to deduce the horizontal and vertical displacement of the user (in meters) and headset rotation (yaw, pitch and roll).

5.2.3. Questionnaires

Four questionnaires were distributed to participants: socio-demographic, traditional drawing skills, VR skills, mental rotation test (MRT) and usability questionnaire.

Socio-demographic: This questionnaire included personal details: age and gender.

Traditional drawing skills: To assess the drawing skills, we asked the participants the following questions: did they have any training in traditional drawing (hobby or professional); how often did they use drawing (never, few, sometimes, often, very often); to evaluate themselves subjectively on their drawing skills, that is, their comfort in using drawing (not at all comfortable, a little comfortable, moderately comfortable, quite comfortable, completely comfortable). Clusters were created according to the answers to the three questions. For this, training is rated from 0 to 2 points (1 point for hobby training and 2 points for professional training); frequency is rated from 0 to 4 points; fluency is rated from 0 to 4 points. Then, participants were divided into groups of level (novice, apprentice, advanced, expert) using k-means.

VR drawing skills: VR expert versus VR novice: To assess the VR skills, we asked the participants the following questions: did they have any training in VR (yes: VR expert, no: VR novice); have you ever used VR sketching (yes, no). All the VR experts have already used VR sketching and all the VR novices have never used VR sketching.

Mental rotation test A (MRTA): The MRTA (by Peters et al. 1995) is a redrawn version of Vandenberg & Kuse’s (1978). The test has 24 items organized in 4 pages. Each item is composed of five figures: a reference model on the left side and four figures located on the right side of the reference model among which the participants have to indicate the ones that are similar to the reference model. There are always two correct answers per item. The time is divided into 2 × 3 minutes with a 4-minute break in between. One point is given per item if the participant finds the two correct figures. No points are given if the participant finds 1 or 0 figures. The sum of these points will give the MRT score ranging from 0 to 24.

System Usability Scale (SUS): This 10-item survey aimed at recording subjective assessments of usability (Brooke 1996; Lewis & Sauro 2009) is a “quick and dirty” tool with five response options from strongly agree to strongly disagree. We used the French-validated version (Gronier & Baudet 2021). The SUS score ranges from 0 to 100. The closer the score is to 100, the better the perceived usability.

5.3. Procedure

The average duration of this experiment was around 60 minutes and was structured in four steps: (1) participants were asked to complete a series of questionnaires (socio-demographic, traditional drawing skills, VR skills, MRT); (2) then the participants had to perform task 1 (traditional sketching), that is, to reproduce a writing-desk using a pencil and paper in 7 minutes. They were asked to reproduce as closely as possible the photo presented on a computer. They could look at the photo in front of them as many times as they wanted; (3) participants were immersed in a neutral virtual environment (hangar) and were trained to use Time2Sketch. There was no time limit for them to learn the software. Once they were familiar with the software, they had to perform the sketching tasks in VR. Then, the photos of the pieces of furniture to be reconstructed in 3D appeared in the immersive environment. The two VR drawing tasks were presented randomly and consecutively to the participants, who could look at the image for as long as they liked without having to remove the headset. Participants had 10 min to make the basic task (the shelf – task 2) and 20 min for the complex task (the buffet – task 3). The instructions imposed were always the same: “reproduce the furniture as faithfully as possible, taking into account the volumes” including the life-size of the furniture. These three pieces of furniture (Figure 1) have been selected according to their complexity. The writing desk (task 1) has been selected because it requires an effort on the perspectives. The shelf (task 2) is a simple task with a simple geometric shape, while the buffet is a complex task requiring to take into account the opening angle of the cabinets and drawers and many details. This choice of items of furniture (with details, door opening, etc.) makes it possible to reflect all the range of drawing skills of the participants in line with the evaluation criteria. (4) Finally, the participants were asked to answer the SUS questionnaire.

Figure 1. Presentation of the three drawing copy tasks. In task 1, the writing desk is copied with a pencil/paper. The basic task 2 (the shelf) and the complex task 3 (the buffet) are copied using Time2sketch software in a VR environment.

5.4. Measuring quality of the sketches

Two expert in VR drawing judges evaluated independently each sketch (VR sketches are presented under four faces: top, profile, front and ¾ frontal – see Figure 2) with a set of criteria using a grid detailing each point of each criterion (Supplementary Material 1): respect for volume/perceptive; respect for proportion; quality of the lines; fidelity with the original picture. Each criterion is scored from 1 to 5. The sum of the points gives a score between 4 and 25. The higher the score the better the quality of the sketch.

Figure 2. Example of sketches performed by two participants (24 and 25) according to the three tasks: task 1 (traditional sketching), tasks 2 and 3 (VR sketching).

5.5. Data analysis

Results were analyzed using SPSS version 22 (IBM Corporation, 2013), JASP Team (2022) version 0.16.4 and RStudio. Each score task was systematically compared to user characteristics and usability components. Bivariate correlations and ANOVAs were performed when the sample met the homoscedasticity criteria. K-mean was used to perform traditional drawing-level clusters.

5.6. Inter-judge reliability

Two expert judges analyzed the sketches using a grid with four variables (volume, proportion, quality of the line and fidelity). We used Intra-Class Correlation (ICC2,k) two-way random to verify inter-judge reliability for the quality of the sketches (Shrout & Fleiss 1979). The mean ICC2,k measurement for task 1 (the writing desk) was .946 with a 95% confidence interval of .884 to .947 (F(30,26.8) = 19.8, p < .001). The mean ICC measurement for task 2 (the shelf) was .935 with a 95% confidence interval of .745 to .976 (F(30,7.32) = 22.7, p < .001). The mean ICC measurement for the task 3 (the buffet) was .894 with a 95% confidence interval of .664 to .958 (F(30,9.57) = 12.8, p < .001). All criteria above 0.9 are considered excellent, above 0.7 good and above 0.5 moderate (Koo & Li 2016). The reliability has been evaluated by an “average k ratings” that is why we used the averages of the data of judges 1 and 2 for the results.

6. Results

Figure 2 shows some examples of sketches created by the participants. The details of the quality score of the sketches given by the two judges are presented in Supplementary Material 2. Table 1 shows the details of the user characteristics according to the quality score of the sketches for the three tasks.

Table 1. Descriptive analyses of user characteristics, sociodemographic, drawing skills, VR skills, mental rotation skills and SUS score according to the quality of the sketches

6.1. Traditional drawing skills

Traditional drawing training impacts significantly the quality of the traditional sketches (task 1: F(2,30) = 6.187, p = .006, η ² = 0.31) and the quality of the VR sketches (task 2: F(2,30) = 9.239, p < .001, η ² = 0.4); task 3: F(2,30) = 7.39, p = .003, η ² = 0.345). Participants who received training in traditional drawing produce much higher quality traditional and VR sketches than those who have not received training. Similarly, the subjective evaluation of these traditional drawing skills (comfort in drawing) is not significantly linked to the quality of traditional sketches (task 1: F(3,30) = 2.328, p = .095) and but is significantly linked to the quality of VR sketches (task 2: F(3,30) = 4.86, p = .008, η ² = 0.35, task 3: F(3,30) = 6.93, p = .001, η ² = 0.44). Participants who perceived themselves as good traditional drawers drew better than those who perceived themselves as poor traditional drawers. The frequency of use of the traditional drawing does not impact significantly the quality of the traditional sketches (task 1: F(2,30) = 0.72, p = .49) and the quality of the VR sketches (task 2: F(2,30) = 0.485, p = .621; task 3: F(2,30) = 0.048, p = .95).

Based on the three variables (training, comfort and frequency), four clusters were performed with k-means R ² = 0.743 (novice: mean = .36, SD = 0.67; apprentice: Mean = 1.9, SD = 0.74; advanced: mean = .4.18, SD = 0.98, expert: mean = 7, SD = 0). Clustering of traditional drawing skills impact significantly the quality of the traditional sketches (task 1: F(3,27) = 3.54, p = .028, η ² = 0.28) and the quality of the VR sketches (task 2: F(3,27) = 4.61, p = .01, η ² = 0.34); task 3: F(3,27) = 4.66, p = .009, η ² = 0.34).

6.2. VR drawing skills

The VR experts performed significantly better quality of VR sketches than VR novices (task 2: F(1,30) = 6.61, p = .016, η ² = 0.19; task 3: F(1,30) = 6.82, p = .014, η ² = .19) but there is no significant difference for the traditional sketching (task 1: F(1,30) = 0.175, p = .68).

6.3. Mental rotation skills

There was no significant correlation between the mental rotation score and the quality of the traditional sketches (task 1: r = 0.34, p = .063). Conversely, the mental rotation score is significantly correlated (positively and strongly) with the quality of the VR sketches (task 2: r = .617, p < .001; task 3: r = .52, p = .006). The higher the participants’ mental rotation score, the higher the quality of the VR sketches.

In addition, there was no significant correlation between the mental rotation score and the VR skills (F(1,29) = 1.255, p = .272).

6.4. Movements in VR sketching

There was no significant correlation between the position in the scene and headset rotation and the quality of VR sketches for task 2 (horizontal movement: r = 0.11, p = .55; vertical movement: r = 0.05, p = .79; yaw: r = −0.18, p = .34; pitch: r = 0.087, p = .64; roll: r = 0.2, p = .28). There is a significant correlation between movements in the scene and the quality of VR sketches for task 3 (horizontal movement: r = 0.47, p = .008; vertical movement: r = 0.48, p = .006) but the correlation is not significant with the headset rotation (yaw: r = 0.1, p = .6; pitch: r = 0.26, p = .15; roll: r = 0.29, p = .16). Participants who moved more in the scene had better quality complex VR sketches. Conversely, head movements that created parallax did not improve the quality of the sketches.

There was no significant difference between the movements and headset rotation in the scene and the VR skills (task 2: horizontal movement: F(1,29) = 0.015, p = .9; vertical movement: F(1,29) = 0.03, p = .86; yaw: F(1,29) = 0.152, p = .7; pitch: F(1,29) = 0.52, p = .48; roll: F(1,29) = 0.11, p = .75; task 3: horizontal movement: F(1,29) = 0.9, p = .35; vertical movement: F(1,29) =0.79, p = .38; yaw: F(1,29) = 0.3, p = .59; pitch: F(1,29) = 0.61, p = .44; roll: F(1,29) = 0.07, p = .79) and traditional drawing skills (task 2: horizontal movement: F(1,29) =0.72, p = .49; vertical movement: F(1,29) = 1.1, p = .35; yaw: F(1,29) = 0.23, p = .8; pitch: F(1,29) = 0.3, p = .74; roll: F(1,29) = 0.42, p = .66; task 3: horizontal movement: F(1,29) = 0.47, p = .63; vertical movement: F(1,29) =0.37, p = .69; yaw: F(1,29) = 0.044, p = .96; pitch: F(1,29) = 0.48, p = .62; roll: F(1,29) = 0.29, p = .75).

There was no significant correlation between the movements and headset rotation in the scene and the mental rotation skills (task 2: horizontal movement: r = −0.01, p = .95; vertical movement: r = −0.08, p = .97; yaw: r = −0.22, p = .25; pitch: r = −0.08, p = .68; roll: r = −0.09, p = .61; task 3: horizontal movement: r = −0.04, p = .61; vertical movement: r = −0.04, p = .84; yaw: r = −0.15, p = .44; pitch: r = 0.02, p = .9; roll: r = 0.11, p = .54).

6.5. System Usability Scale

Participants gave an average SUS score of 75.57 (SD = 12.68, range = 50–97.5). There is no significant correlation between the SUS score and the quality of VR sketches (task 2: r = 0.14, p = .45; task 3: r = 0.15, p = .42), and no link is observed between the SUS score and the VR skills (F(1,30) = 0.001, p = .98) and the traditional drawing skills (traditional drawing training: F(2,30) = 0.02, p = .98; frequency of using traditional drawing: F(2,30) = 0.3, p = .74; comfort in traditional drawing: F(3,30) = 0.77, p = .52) made with the software Time2Sketch.

7. Discussion

This study’s objective was to better understand the skills needed for traditional sketching and VR sketching (with Time2sketch) for non-designers. To do this, we collected drawing skills, VR drawing skills using a questionnaire, mental rotation using the MRT, and two independent judges rated the quality of the traditional and VR sketches of the participants. We made four hypotheses in which traditional drawing skills, VR drawing skills, mental rotation and movement would have an impact on the quality of traditional and VR sketches.

Our first hypothesis was that participants with high traditional drawing skills would create a better traditional and VR sketches than those with low traditional drawing skills. Results validate our first hypothesis. Three variables were measured to assess the traditional drawing skills: traditional drawing training, frequency of use of the traditional drawing and comfort in traditional drawing. Our results show that participants who received training (hobbies and professional) and who rate themselves positively produce significantly better traditional and VR sketches. These results are in line with those of La Femina et al. (2009), McManus et al. (2010) and Chamberlain et al. (2015). However, the frequency of use does not impact the quality of traditional sketches. Nevertheless, the data used are not very diversified. For instance, there are only six participants who have received training, none of whom consider themselves to draw often or very often. To limit this bias, we clustered this skill into four groups (novice, apprentice, advanced and expert) based on these three variables (training, comfort and frequency). The results show that there is a significant difference between the groups and the quality of traditional and VR sketches. More precisely, the better the assessed level of traditional drawing skill (e.g., expert or advanced) the better the quality of the traditional and VR sketches and vice versa.

Our second hypothesis was that VR experts will have a better quality of traditional and VR sketches than VR novices. Results validate partially our hypothesis. VR experts performed better quality of VR sketches than VR novices, but this is not the case for traditional sketching. These results are in line with those of Wiese et al. (2010) and Bolier et al. (2018). Because of the emergent nature of VR technology, there is not yet a cohort of designers trained in VR drawing that we could have used in this study. We were therefore more interested in the expertise in VR. We can wonder about what is a VR expert and when does one becomes an expert? To the best of our knowledge, no research is currently being done in this area. We considered that experience and training were the main elements to consider that a person was an expert in the field. Conversely, novices rarely interacted in a virtual environment. We could have provided more nuances to these two profiles by questioning the expertise in VR. For this, it would be interesting to carry out studies on the learning of drawing techniques in VR according to the training.

According to the first two hypotheses, the traditional and VR drawing skills are a crucial element in the traditional and VR sketches quality, regardless of the complexity of the task (no difference observed on the effect size between tasks 2 and 3). However, traditional sketching techniques seem transferable to VR sketching but not vice versa. One of the requirements for using VR sketching for novices would be strong traditional drawing skills, that is, have previous training in traditional drawing or feel comfortable with traditional drawing.

Our third hypothesis was that the higher the visuospatial abilities of the participants, the higher the quality of the traditional and VR sketches. We focused on mental rotation to assess the visuospatial abilities using the MRT (Vandenberg & Kuse 1978). Results partially validate our hypothesis. A high mental rotation score is related to high quality of VR sketches but not linked to traditional sketches quality. These results are in line with Barrera Machuca et al. (2019) but not with Alias et al. (2002). Alias et al. (2002) showed an impact of visuospatial abilities measured with the SVATI – Spatial Visualization Ability Instrument (Embretson 1997) which is highly correlated with the Vandenberg MRT (Alias 2000) on the quality of the traditional sketches. Conversely, Barrera Machuca et al. (2019) found that the user’s visuospatial abilities (using the vz-2 paper folding test – Ekstrom, French & Harmon 1976 – and the perspective taking/spatial orientation test – Kozhevnikov & Hegarty 2001) affects the shape of the sketches, but not the line precision in the VR sketching. In line with Cohen’s (2013) guidelines, we observed a difference in correlation strength between task 2 (strong correlation) and task 3 (moderate correlation) which suggests that mental rotation skills would not be related to accuracy since task 3 requires more accuracy because of the number of details. If the complexity of the VR sketching task increases, the training and the traditional drawing strategies (i.e., traditional drawing skills) would compensate for the mental rotation skills. On the one hand, visuospatial abilities could help the draftsman, but a training is required to achieve a good quality of traditional sketches. On the other hand, mental rotation skills seem to be an essential requirement for the basics of VR sketching. However, accuracy is one of the important challenges that VR sketching has to face (Wiese et al. 2010; Arora et al. 2017; Barrera Machuca et al. 2019). The lack of accuracy is detrimental to the creation process because the sketch may not correspond to the user’s intention (Barrera Machuca et al. 2019).

Our fourth hypothesis was that participants who move the most will have a better quality of VR sketches. After analysis of the movements recorded with the headset, we can partially validate our hypothesis. The movements impact the quality of the VR sketches only for task 3 and only for position in the scene and not the headset rotation. Thus, as with the previous hypothesis, the level of complexity of the task seems to impact user behavior and requires more spatial inspection and movement. If task 2 does not require movement to improve the quality of the VR sketches, it is probably related to the size of the task which required little backward movement and spatial cues. Indeed, task 2 is a square tube whereas task 3 includes different levels based on different sizes of rectangles. In addition, details such as open drawers or open doors added the consideration of angle, which is not required in task 2. These results are in line with Barrera Machuca et al. (2019) and Yang & Lee (2020). In addition, we observed that mental rotation skills, traditional and VR drawing skills are not related to spatial inspection. The visual knowledge we have is derived from a two-dimensional (2D) image of an object in a scene on the retina (Frith & Law 1995). Movements and spatial inspection provide information on the depth as well as the previous knowledge helps to prospect on the size and the shape of the object (Frith & Law 1995), that is why, the movement is an important variable to take into account, especially in VR sketching. It seems that the rotation of head is not sufficient to allow the perception of this depth. Nevertheless, it is not known if encouraging users to move would increase the quality of the VR sketching.

Our fourth hypothesis was that participants who perceive the usability of the software as good will have better VR sketches quality than those who perceive the usability of the software as bad. The results invalided the hypothesis. It is important to highlight that regardless of traditional and VR drawing skills and visuospatial abilities, users uniformly rated the perceived usability of the Time2Sketch software. The SUS score (measuring perceived usability) does not have a significant impact on the quality of VR sketches. This suggests that they did not report being bothered by the usability of the software. The mean SUS score was 75.57 (SD = 12.68), which is “satisfactory” (Lewis & Sauro 2009). It has been known for a long time that usability is a major factor of technology acceptance (Davis 1989). However, according to our results, perceived usability of theVR tool is not a determinant of sketching performance. If the software does not have an impact on the perceived usability, it seems that the problem comes mainly from the 3-dimensions. Users are able to use all the features of the application easily but it is their graphic skills that prevent them from having better quality sketches. Figure 3 shows a summary of the results. These data are related to a study done on the same software (Fleury et al. 2022).

Figure 3. Schematic representation summary of the results (task 3) *p ≤ .05; **p ≤ 0.01.

We propose six perspectives to this study. First, the translation of a 2-dimensional picture into 3-dimensions necessarily implies visuospatial abilities. Participants have different strategies for viewing in 2D versus 3D (e.g., Popelka & Brychtova 2013). Movement can be a way to compensate for a weakness in visuospatial abilities. It would be interesting to force users to move when they use VR sketching. In addition, it would be interesting to observe the type of movements made in order to better understand the strategies used. Second, with the same approach, to better understand the role of visuospatial abilities, it would have been interesting to ask users to reproduce a task already presented in 3D to better understand the impact of visuospatial abilities. Third, it would be interesting to compare the traditional and VR sketches with a same task to see if the quality is maintained between both techniques. Fourth, the next study may examine the impact of VR drawing skills on creativity. For instance, Chan & Zhao (2010) showed in a study with primary, secondary and university students, a correlation between drawing skills and artistic creativity. Yang & Lee (2020) showed that VR sketching allows to be more creative than traditional sketching, but the authors do not take into account the drawing skills and the visuospatial abilities. To our knowledge, no study has shown the impact of VR drawing skills on creativity in the ideation process. Fifth, our results are based only on Time2sktech software. In order to avoid the difficulties experienced by novices, it would be interesting to analyze if they use more easily sketching software adapted to their visual–spatial difficulty, such as software to create 3D blocks or sculptures or with a Hybrid Virtual environment – for example, the Hyve-3D used in the case of architectural co-design – using handheld tablets to manipulate a cursor on a plan (Dorta et al. 2016) or more recently a pen and table interact (e.g., VRsketchIn – Drey et al. 2020). Nevertheless, the advantages of Time2Sketch compared to other devices (e.g., Hyve-3D, VRsketchIn) are its speed execution, its immediate scaling and its flexibility, especially its ambiguity and imprecision allowing multiple interpretation readings which supporting creative leaps (Ullman, Wood & Craig 1990) which is an essential tool for good sketches (Buxton 2007). Moreover, Time2Sketch is a mobile and affordable software since it only requires a VR headset and can be used everywhere. Sixth, last but not least, this study is the beginning of many others that will gradually be integrated into the Do-It-Yourself process in order to test and validate the new process. The ultimate aim will be to enable novice users to create a made-to-measure piece of furniture with the help of an experienced designer.

There are four limits of our study. First, we measured the visuospatial abilities with MRT. However, there are other dimensions of visuospatial abilities (Linn & Petersen 1985) such as mental transformation (Tartre 1990) and visuospatial working memory (Logie 2014). Nevertheless, mental rotation skills are associated with other visuospatial abilities (e.g., Alias 2000; Ault & John 2010). For instance, Muffato, Meneghetti & De Beni (2020) observed a strong correlation between the sMRT (Short Mental Rotations Test), the sOPT (Short Object Perspective Taking Test) and the VSWM (visuospatial working memory). In addition, the MRT is one of the most cited and preferred tests in studies of industrial design education (Kelly 2012). Second, we chose a furniture use case which makes the task less pure than if we had presented volumes without realistic equivalent. In addition, users could sketch on their prior knowledge to help them consider the shape of the furniture (Frith & Law 1995). In the same way, we limited the time for the creation of the task (7 min for task 1, 10 min for task 2 and 20 min for task 3). Some participants were not finished by this time. To respect the real process, end-users will not have a time limit but in an experimental setting, it was necessary to put one. Third, Time2Sketch software will be useful in the case of creative tasks or communication with different users or designers in the early phase of the design process. We wanted to know if novice users in VR would be able to use such software and would be able to communicate on it. We do not have data on the ability of users to communicate though their sketches, this could be the subject of a new study. Then, when creating a sketch in a design process, users use their mental image to create a new and unique production using their mental image which is not the same task as duplicating a stimulus. We therefore have no data on the ability of users to sketch their own mental image of furniture but we would not have been able to evaluate the graphic quality of the mental image because the sketches would not have been comparable between them. We consider that if participants are able to reproduce an image, then they will be able to transfer their mental image. Fourth, we based our data on VR sketching on only on software (Time2sketch). Nevertheless, according to the SUS score, Time2sketch did not seem to disturb the participants. It would be interesting to generalize the results to other VR sketching software.

To conclude, traditional sketching is easier to use than VR sketching – with T2S software – which requires more drawing skills and visuospatial abilities of the novice users. Regardless of the usability of the software, we identified three requirements for the use of VR sketching: first, it is required to have previous experience in traditional drawing or VR. Second, it is required to have high visuospatial abilities. Third, it is strongly advised that users move around in the virtual environment during a VR sketching task to become aware of the depth of the drawing in progress. We recommend that novice users to use mainly traditional sketches in order to express their needs and avoid misunderstandings with the future designer or to use another sketching software in order to suggest a software more adapted to their needs (e.g., using 3D blocks or Hyve-3D).