Design team formation using self-assessment and observer-assessment techniques: mapping practices in a global network of universities

Abstract

Design engineering education is increasingly challenge-based, which requires educators to form cohesive student teams capable of delivering desired outcomes while fostering learning and collaboration. An example is an international network in which students from different global universities collaborate. Student teams work on researching the problem space, re-framing their challenge and producing multiple prototypes. The challenge for the teaching teams is to be able to form multiple cohesive teams out of a pre-selected group of highly motivated students. Because of the exclusive nature of this educational program, it is a suitable case study for exploring student design team formation practices. The aim is to identify the methods, tools, theoretical underpinnings, challenges and limitations of student team formation. We interviewed teachers from seven universities about their practices. The interviewees had several years of experience in team building. The interviews were analyzed to contrast practices across universities as well as to the team formation literature. Our findings show that mixed methods that combine self-assessments and observer-assessment methods are the preferred means of forming teams. Our findings also show that current practices have evolved over time through trial and error, and are only partially grounded in different literatures and not necessarily in team formation literature.

1. Introduction

Higher education institutions have long adopted collaborative teaching methods where students work in teams towards a common educational objective, both because of the benefits it presents to the student learning process and to educate professionals that go on to be capable of collaborating with other professionals in working environments. Among these collaborative teaching methods used in design and engineering schools, challenge-based teaching has gained popularity (Okudan & Adams 2003) given not only the possibility that students have to collaborate towards a common learning objective but also towards the resolution of a real-world problem.

For all the benefits presented by a challenge-based teaching/learning approach, it does come with its fair share of difficulties. Any student who has participated in education tasks where they are teamed up with other students has struggled with the difficulties of collaborating with others. To paraphrase Dickens (1859), working in student teams can be the best of times and it can also be the worst of times. Well-performing teams, where skills and capabilities are well distributed, where teammates are truly cooperative and where there is good chemistry between teammates, can enhance both the learning, as well as allow the production of innovative solutions to the proposed challenges. While a poorly constructed student team might, on the other hand, lead to in-fighting within the team, poor performance in the resolution of the challenge, different levels of production and commitment (students not putting in their fair share of work) or the team deciding to partition the work without collaboration with one another. While the eventual performance of a team depends on much more than just the initial team formation, this starting point of a project is obviously important, and thus the focus of this paper.

In this study, we conducted semi-structured interviews with teaching teams from a network of universities that work on challenge-based design problem solving and innovation, with the intention of examining the different methods and approaches used by these experts on team formation. Not with the intention of presenting a “how to” guide to team formation but rather to answer the research question of what the methods, tools, theoretical underpinnings, challenges and limitations of educational team formation are and to discuss current practices and lessons learned.

1.1. Challenge-based projects and team formation

For this paper, we interviewed teachers from seven universities around the world who have jointly created and participated in projects that utilize design problem-solving methodologies and a challenge-based approach to education. The SUGAR network is a global network of more than 25 different universities where students work in student teams on product design and development projects. In each project, a multi-disciplinary group of 3 to 5 students from one university is teamed up with a similar multi-disciplinary group of 3 to 5 students from a second university (global team) along with a corporate sponsor that provides a challenge to be solved. Over the course of roughly 9 months, the global teams work on researching the problem space, re-framing their challenge and producing multiple prototypes of design concepts.

The challenge for the teaching teams of the participating universities is to be able to form multiple cohesive teams out of a pre-selected group of highly motivated students who apply to participate in the course. While at the same time performing a balancing act of making sure that the project objectives towards the participating corporate sponsors are being met and ensuring that the participating students’ education and learning experience remains at the forefront. Hence, this context is distinct from regular courses, with usually larger batches of students, and concerns such as free-riders and slackers.

In striving to form diverse and cohesive teams, teaching team members assess and use multiple variables or criteria about the students with different degrees of control to form the teams. Among them are skill and background variables, personality and behavior variables, and diversity variables. The skill and background variables can be directly related to the students’ field of study (different types of students from design, engineering and management among other fields), their level of expertise in these fields and/or other skills acquired by the student in other settings that could be beneficial for the given challenge. The personality and behaviors variables relate to among others, the students’ attitudes and behaviors while collaborating with others and/or the roles they assume in teamwork. Finally, in the diversity variables, aspects such as age, nationality, gender and languages spoken are taken into consideration based on the challenge at hand. However, there are other sets of variables that are outside of the individual teaching team’s control that can also play a role in team forming, such variables can be related to the partner universities teams, as different collaborations might require different skills and other variables can be introduced by the corporate sponsor.

To be able to cope with the challenge of forming these teams, teaching teams from different universities have developed their own methods for team formation. These methods have been honed by multiple years of experiential learning (March 2010), small-scale experimentation of trial and error and the usage of different theoretical approaches from different fields of study (management, design, psychology, decision making, etc.).

2. Team formation background and approaches

2.1. Forming teams

There are several factors that are important in the process of team formation, and for which we can draw lessons from the existing literature. On the one side, what are the different elements and criteria to take into consideration when forming a team that will perform well in the resolution of the given challenge. On the other hand, how to best recognize–spot–diagnose said characteristics in the individuals that are taken into consideration to form the teams. And finally, which configuration of these characteristics is more desirable. In terms of which criteria to take into consideration, there is only limited disagreement in the literature, yet for the latter two, how to recognize the criteria and how to utilize them, we can find more dispersion in methods and results.

The first factor, which criteria to take into consideration, shows the most unanimity in the literature on educational team formation. While professional skills and expertise remain important, personality and behavioral skills remain among the most important elements. Characteristics such as communication skills, effective interaction, project management skills, self-efficacy, autonomy and positive interdependency among others, have a great effect on the team’s ability to properly function (Okudan & Adams 2003; Potosky & Duck 2007; March 2010; Estes et al. 2013; Kiernan, Ledwith, & Lynch 2020)

2.2. Approaches to team formation

When trying to answer the questions of how to recognize the desired criteria in students and how to utilize them to form teams, literature offers a wide range of methods and approaches. To start, it is important to note that it is not uncommon to find educational team formation methods that do not involve the recognition of criteria or an intelligent method to group students. These methods include randomized grouping or self-selection of teams, where students choose who to work with. These methods have been shown to be less effective for good team performance (Estes et al. 2013). On the other hand, there are purposeful methods and approaches to team formation that attempt to measure certain criteria and then utilize those to create teams, which are more relevant to this paper. Based on the explored literature that relates to team formation, we can find that in the purposeful approaches to team formation, the process of recognition of the criterion is performed in either of two methods, self-assessment and observer-assessment (see e.g. Tumpa et al. 2022).

Self-assessment methods are based on the use of tools where the participants reply to a series of questions and based on the answers certain criteria can be determined. We can find emphasis in the literature on team formation on these types of methods, which utilize tools that are derived from the characterization of personality traits and behavioral skills (Mcclough & Rogelberg 2003; Wilde 2008; Kress & Schar 2012; Estes et al. 2013; Rodríguez Montequín et al. 2013). Among these types of approaches, the most commonly used are big five personality factors, Myers-Briggs types of indicators (MBTI) and other derivations from MBTI such as Wilde’s teamology. On the other hand, though less commonly studied in the literature, there are observer-assessment methods. These methods are based on the use of activities designed to reveal behaviors and skills of the students through practical exercises and interactions with others. During the execution of these activities an observer, usually the same teacher(s) administering the activity, observes the participants to attempt to recognize the desired criterion (Potosky & Duck 2007; Luojus et al. 2017). Such methods are among the most utilized in project-based/challenge-based engineering and design education and are the subject of this paper.

Although self-assessment methods for team formation are used in educational team formation, they come with their share of shortcomings. First, self-assessment methods rely heavily on the answers provided by the participants to create the personality construct that then are used to create teams. However, this raises issues of validity of the results obtained based on possible misrepresentation, self-deception, or purposeful lying (Paulhus 1986; Mount, Barrick, & Strauss 1994; Holden & Hibbs 1995) with the intention to either impress or hide undesirable attributes from the evaluator by the responder. Second, in educational settings, it is not uncommon to find that in groups of students from a similar background, there is more possibility for homogenous results on the personality constructs to appear (Kress & Schar 2012). This homogeneity of constructs then limits the combination of possible team constellations that can be constructed. Third, the use of tools for cognitive and behavioral attribute identification does not measure compatibility and effective interaction between participants, which is shown to be a more significant in team performance (Potosky & Duck 2007; Bergey & King 2014). Finally, of the most well-known and utilized method for educational team formation, Myer Briggs, further questions of validity and empirical grounding have been raised (Pittenger 1993; Stein & Swan 2019). Several studies mentioned by Pittenger (1993) show that the MBTI has low reliability and validity, meaning that it may not provide accurate or consistent results over time or across different populations. Furthermore, the issue of categorization and stereotyping created by these limited type-structure tends to oversimplify “complexities of human personality into an artificial and limiting classification scheme” (Pittenger 1993, p. 51) which may put into question its reliability as a single source for team creation.

That leaves observer-assessment as a strategy, where the observer is in this case the teacher composing the teams. While there is some literature (Wenninger 2019; Tumpa et al. 2022) on assessing team performance in light of summative and formative assessment (i.e. feedback and grading), using observer-assessment techniques in the team formation phase is under-addressed in the literature.

3. Methology and data collection

To better understand the process of team formation in design and engineering project-based education a group of 10 teachers from 7 different universities, who participated in the previously mentioned SUGAR projects, were interviewed. The group of teachers selected to be interviewed all had multiple years of experience in forming teams of students that participated not only in the projects related to SUGAR but also in multiple challenge-based design and engineering courses. The sample included TT members with backgrounds in business, design and engineering. The student groups they work with include at least various engineering programs, but in many cases also include students from other programs such as business and design. The number of students assessed by the different teaching teams at each university ranged from 15 students to 100 students per year and the number of teams created also ranged from 2 to 10 teams per university per year, greatly depending on the number of open spots and projects being run by each university. SUGAR currently has some 25 participating universities, but not all have multiple teams per year, nor teaching team members with several years of experience, thus reducing the number of potential interviewees.

The set-up of SUGAR, putting high requirements of students and teachers, and with the extra challenges presented by the international collaboration and the multi-disciplinary composition of student teams, make it especially suitable for studying team formation practices.

3.1. Data collection

In the process of interviewing the different teaching teams, a semi-structured interview method was used. Each interview had an average duration of 60 min, and during the interview, the participants were asked to recount and explain the process that they followed during their team formation process. The semi-structured interviews used a question protocol that is provided in Appendix A.

To conduct the interviews a semi-structured interviewing protocol was developed with three main areas of interest to the interviewers. The first area relates to the activities used by the teaching teams in the process of team formation and how these activities were executed. The second area relates to the different criteria (behaviors and skills) being observed during the different activities. Finally, the third area is related to the process used to create teams using the observed criterion in the different activities. However, the process of interviewing the participants did not follow a sequential order of these three areas. Instead, the recount of the process of team formation of each teaching team was used as the chronological guide, and then the interviewers used the protocol as a guiding tool to ask follow-up questions that pertain to the three areas of interest.

In the final part of the interview, the participants were also asked to reflect on their own process beyond the simple execution. This led to conversations of evolution of their team formation process through the years and their own reflection and evaluation of their team formation process.

3.2. Data analysis

All interviews were conducted digitally, given the different geolocation of the participants, and recorded for analyzing purposes. These recordings were transcribed, and qualitative coding was used to process them and in combination with the previously developed protocol, they were codified into the three areas of interest in order to contrast the practices at each university to each other as well as to the existing team formation literature. Because the interview questions were directly related to the research question, and the number of interviews was manageable, this was a relatively straightforward process. Furthermore, this codification also allowed to encounter repeating themes in the practices of these universities that help to better understand observer-rating based team formation process and how it differentiates from self-rating team formation tools. These themes were (1) connoisseurship and expertise, (2) small-scale experimentation and (3) experiential learning. The use of themes in this study follows in a similar line as Crilly’s (2015) thematic analysis of expert designers and fixation in which the themes are used as a discussion tool that is meant to represent the voice of the participants.

4. Findings

4.1. Contrasted practices

Initially teaching teams that were interviewed were deemed to have observer-assessment methods as their primary method for team formation. However, after the interviews took place, it was revealed that most of the teaching teams utilized mixed methods that combined the use of activities that integrated self-assessment as a guide for skills and behaviors to be explored/checked later in observer-assessment-based activities. Even though every teaching team had its own particular activities and methods they followed a very similar pattern of process in which self-assessment and observer-assessment activities were used to finally form teams (Figure 1). We present a contrast between the different teaching teams in three key areas; the activities they use to recognize the criteria that are used in their team formation process, the criteria used to characterize the participating students, and finally how they use said criteria to form teams.

Figure 1. Visual representation of team formation process pattern from interviews.

4.1.1. Criteria recognition activities

In the interview process, each teaching team was asked to recount their process for team formation, and special emphasis was put on the activities they used for this purpose. Each teaching team had a particular set of activities that they used and differed in particular from each other, but most had a very similar aim and rationale behind them.

Most processes began with an application/selection process that was also used as an initial point for recognition of attitudes and behaviors from candidates. These early moments of the team formation process, the teachers explained during the interviews, served as an initial contact with the students that helped to map a priori assumptions of the students’ possible skills and behaviors. Typical activities used by the teacher during this stage were curriculum vitae (CV), application letters, psychological questionnaires and interviews with the candidates. So, at this stage, most teams rely on self-assessment methods.

Following the initial mapping of the students and their skills and behaviors, some of the teaching teams went on to a stage of “corroboration” of those initial findings. Teaching teams then explain that in this second stage, some specially designed activities were used, in which the students engage in the solving of a task that requires the use of typical skills needed during the projects. These tasks were normally oriented towards the solving of design tasks with different levels of complexity and time. Tasks such as redesigning an existing object, building prototypes for specific users or the use of a common task called “the paperbike challenge” (see below), known to all teaching teams of the SUGAR network, are used. While the students performed said tasks, the teachers engaged in deep observation of behaviors and interactions among the students and then these observations were used in the final step of team creation.

Based on the information and observations gathered in these previous stages, teachers then moved to the final team-forming activity/deliberation, which in all cases did not include the student’s participation any further. The complete team formation process had a big variation from one teaching team to the next. In some cases, the process would be comprised of multiple activities that spanned several months while others used a couple of activities in the course of one day.

4.1.1.1. Activities

In the interviews conducted, most teaching teams explained the particular activities that they utilized in their team formation process and the rationale behind them. Among them, we could find a couple of salient examples that provide a picture of said practices.

Design tasks: One of the most common exercises utilized by the teaching teams to reveal certain characteristics from the participants had to do with the resolution of a design task. Tasks such as redesigning commonly known objects (wallets, backpacks, bathrooms) were given to the students (Figure 2). As recounted by the teaching teams, the design tasks served multiple purposes. They can be used to either observe technical skills and the use of common prototyping methods (Figure 3). However, the main purpose was to observe the students’ behaviors and interactions in a relaxed environment where the students weren’t predisposed to being observed.

Figure 2. Results produced by students from different design task assessments.

Figure 3. Results produced by students from different design task assessments.

In another case, the design task was also used to observe how participants dealt with ambiguity and uncertainty, two major components of the course. In this case, students were given little or no instruction about the design task, and while they tried to solve the task and build prototypes, teachers observed their behavior.

Dealing with frustration: As part of the SUGAR projects, one of the common traits of the projects is to deal with failure and frustration in a positive way. Learning to evaluate prototypes not as failures or successes but rather as learning experiences even if said the prototype was deemed “unsuccessful”. With the intention of observing student behavior on the face of an impossible task, one team uses a puzzle exercise that becomes increasingly more difficult until the last level where a special piece is created to make the puzzle unsolvable (Figure 4). This allows the teaching team to observe how the students behave and interact with each other when they are frustrated and when the task at hand will, in the case of this exercise, inevitably fail (Figure 5).

Figure 4. Unsolvable puzzle meant to observe behaviors from students in an unsolvable task.

Figure 5. Students interacting during the unsolvable puzzle task.

Assessment dilemmas: Another example of activity that is used by the teaching teams to observe behaviors and characteristics from the students is the use of classical assessment exercises that present students with dilemmas in which they have to discuss, negotiate and reach agreements. In the cases of exercises, the group of teachers used the help of a third party or mediator (university psychologist) to guide the session in which the students are present, for example, with a scenario of survival in an end-of-the-world scenario in which they must discuss and make a series of decisions in order to survive. The main intention of said assessment is to observe interactions and behaviors rather than whether the objective of surviving is accomplished or not.

Paperbike: Finally, one of the traditional tasks that are performed in the frame of the SUGAR network is called the paperbike or the paper challenge. The exercise is meant to be a warm-up activity in which teams of students build a vehicle made of paper and cardboard and must compete in a series of games and obstacle courses (Figures 6 and 7). The two main intentions of the activity have been to introduce the design methodology in which the courses take place in a small-scale challenge that is unrelated to the project and to start the process of familiarization between the participating students. However, some teaching teams had an extra objective, very similar to those of the design tasks. At least two universities state this activity allows the teaching teams to observe students interacting with each other while they build their vehicle, which allows them to observe behaviors, interactions and also bonds being created among certain students (Figure 8).

Figure 6. Students during the building process of the PaperBike exercise.

Figure 7. Students during the building process of the PaperBike exercise.

Figure 8. Final assembly of a paper bike by a group of students.

4.1.2. Criteria

During the process of team formation and the different activities undertaken by teaching teams, they have one objective in mind, to identify different criteria in the participating students that then guide the formation of the teams. From the interviews, it was observed that the teaching teams observed three different types of groups of criteria and that each type equally informed the final creation of teams. The three groups are (1) professional and practical skills, (2) behavioral and personality and (3) interactions with others.

The projects in which the students participate are commonly guided by the type of challenge that the sponsoring organization provides. Different challenges require very different skills that are commonly attached to different university programs. In practical terms this might mean that based on the specific challenges the team might need a set of technical skills such as programming, digital modeling, retail operations and medical imagery, to name a few examples. Based on this, teaching teams establish which set of professional skills might be needed based on the project’s specific prompt.

These lead to the first group of criteria, professional and practical skills. Teachers commented that the first approach to the team formation process is connected to what type of skills are needed for the challenges and which skills the students have. This group of criteria was more commonly evaluated by activities that require the student to self-report them, such as CVs or cover letters. However, in the case of available third-party observations, such as past teachers’ assessments of those skills were highly valued.

Following this first set of criteria, the next two groups related more to behavior and personality traits that the participating students exhibited and how they interacted with others. Teachers reported that traits and skills such as listening to other perspectives, managing conflicts, regulating emotions, taking and learning from feedback, collaborating with others, managing frustration and active engagement were among the most important elements taken into consideration when selecting students to participate in the teams. These two sets of criteria, as described during the interviews, played a crucial role in how teams are formed and how they perform. Teachers argued that in many cases, these criteria had a bigger impact on their decision making. Conflictive teams that are uncooperative and lack good dynamics and communication were discussed as less desirable than teams with more homogeneity in skills and abilities. Teachers discussed during interviews, that in order to recognize and identify behaviors and interactions, observer-assessment such as interviews, design challenges and one-on-one interactions were preferable. These types of activities provided teachers the opportunity to early in the process recognize possible dysfunctional pairings and avoid them while creating teams. As examples of these, teachers mentioned caution at pairing several strong-minded and vocal people who would jockey for team leadership which can create a negative working environment and infighting. On the other hand, they also mentioned caution against pairing several quiet, analytical types of students who would avoid being vocal in team decisions, making it difficult for the team to reach decisions due to inaction.

4.1.3. Creation of teams

The final step of the team formation process, as explained by the teaching teams, had to do with the final decision-making process of putting together teams by using what they have learned from the criteria recognition activities. The interviews revealed that the final process of team creation followed a very similar pattern in almost all interviewed teaching teams. Here it is quite important to note, that so far, we have been using the term teaching teams rather than only teachers. Almost all of the interviewed programs in the different universities were comprised of a group of teachers who participated in the team formation activities and contributed to the final creation of the teams.

The subsequent activity that was described during the interview process was often described as a conversation or deliberation between teaching team members to compare and contrast findings and observations during the assessment activities. In some cases, this conversation was described as a validation of common agreement. In other cases, it served to bring to the table cases where further discussion was needed, if discrepancies between observations existed. During interviews it was commented that these discussions were useful to “give a fair chance” to all students, in case one or more teachers had different observations from different points of view or areas of expertise. Furthermore, it was also explained that during this stage, other teachers outside of the teaching team could be consulted in order to provide further information on the students’ behaviors, professional skills and/or performance in past courses.

4.1.3.1. Team formation practices

In the interviews conducted, most teaching teams recalled the use of either tools or methods that they utilized with the aim of helping the team creation process, as distinct from the observation of characteristics. Among them, we could find a couple of salient examples that provide a picture of said practices.

Trial teams: In many cases, the teaching teams reported that as part of their practice, they used trial teams or experimental configurations that allowed them to start creating combinations of students. The use of trial teams provided an opportunity to create hypotheses of possible combinations. These trial configurations were perceived as a positive measure during the team formation process as it provided more points of observation of student interaction. These “team prototypes” were mentioned to reveal positive configurations where the hypothesis configuration was validated while at other times it revealed team dysfunction and meant that a new configuration needed to be thought of. Furthermore, these trials provided the teaching teams with a platform to discuss final configurations using tangible prototypes of student groupings. The practice of trail teams also had a wide range of meanings. In some cases, due to surplus of time, it meant that warm-up challenges or mini projects that lasted from a week to a month could be used to test how different sets of students interacted with each other. In other cases, due to time limitations, this meant that during a single assessment day with multiple activities, students were reshuffled into different team configurations in each assessment activity. In other instances, where time and resources were more limited, these trial teams were only used as thought experiments on a whiteboard which allowed the teaching team to play around with different configurations and discuss the implications or possibilities of said teams.

Option weighing tools: Other methods that were also discussed during the interviewing process were tools that facilitate the weighing of options based on assigning a numerical value to the different criteria. In such cases, tools such as pre-designed spreadsheets were used to help teachers to rank candidates based on the desired criteria, and the values assigned during the observation process. In such cases, during the observation process, the different participants of the teaching teams assigned a numerical value to their observation and the tool then presented the candidates in a rank based on their particular skills. It is important to differentiate that the intention of the tool was not meant to rank students from “best to worst” but rather to show who, based on the observation, demonstrated more of a characteristic than others. It’s also important to note that the teaching team members reiterated that the tool was also just another tool to aid discussion rather than a final decision-making mechanism.

4.2. Thematic analysis

The thematic analysis was intended to showcase the underlying behaviors that teaching teams seemed to be displaying, that were either implicit and purposeful in their team formation behavior or something that they were engaging in unknowingly. After an iterative process of analysis engaging concepts from team formation literature and the responses provided during the interviews several themes emerged that were relevant to team formation from the observer-assessment approach and the role of these observers in the process. The themes are here presented in no particular order. It is important to also note that the themes here presented, relate to the practices of the SUGAR network, and are connected to the mechanisms that students engage in, which are similar to the processes that teaching teams engage in when building teams. However, these themes are also relevant to practitioners outside of the SUGAR network, as they point to applicable learnings to other types of student team building.

4.2.1. Connoisseurship and expertise

The group of teachers interviewed for this study had in common multiple years of experience teaching multidisciplinary design/engineering projects in which they had been forming teams of students. These many years of accumulated experiences translate into a vast repertoire of lessons learned and development of skills that allow them to better recognize characteristics in students. The many years of building a library of past projects and students assessed helps them to develop expertise that is then used in team formation activities.

The projects for which these groups of teachers build teams are by their own nature highly ambiguous and complex type of projects. During the interviews, teachers repeatedly mentioned the importance of balancing multiple important factors that need to be taken into consideration. Yet typically these factors are not always clear or known before teams need to be formed. The type of corporate partner that will sponsor the project, which students the partner university will contribute or what specific problem the students will face might not be known until after teams have already been formed. This means that teachers rely on the built-up experience to make decisions without knowing the full picture.

You’ve seen their CV, you’ve seen teamwork [activities], you did the interview, you know a bit about the person, but you don’t really know it all. But then when it comes to team forming, to the final forming, it’s kind of a mess, because you don’t have all the information… You really try to put together all of these layers but in the end of the day, it’s the experience that drives [the decisions].

Many times, this built-up catalog of experiences and observations that has been gathered through the years, helps hone the skill that allows them to create teams. However, as stated before, many of these processes of team formation are not undertaken by a single person but rather by a team of teachers. This also allows the team to rely on the expertise of others and the built-up library of each of the teams’ members to avoid biases introduced by single individuals. This also allows for a wider observation, as each team member is able to view the process from their own perspective and help to mitigate potentially failing to observe either desired or undesired characteristics.

I know it is potentially dangerous, that I might have a bias, to students I have already met in other previous courses… So having [names of other teaching team members] that have not met them and are in that case impartial, balances my own opinions and judgements.

4.2.2. Small-scale experimentation

As part of the frame of the SUGAR projects, in which these teaching teams are part and for which they form teams, small-scale experiments or prototypes are widely used as a tool for problem exploration and resolution. Students build prototypes that are subsequently tested in real or simulated environments and that are then used as learning points to inform the next steps of the process. However, this method for problem solving is not limited to the participating students but rather embraced by the teaching teams when forming teams of students. During the interviews, teachers made references to the use of small-scale experiments in their process for identifying skills and behaviors in the participants and also for corroborating possible team constellations.

As explained in Sections 4.1.1 and 4.1.3.1, teachers commonly used specially designated tasks to help reveal skills, behaviors and attitudes from the students that would help inform their process of team creation. However, these activities were also commonly used for prototyping possible student grouping. Based on previously obtained information from applications, interviews or previous exercises, teachers would propose teams that are estimated to have potential, and then said teams are put to the test in said task. Teachers expressed that these team prototypes would then yield learnings about team dynamics and interactions between the students. The learnings from these prototypes are then not only used to test possible constellations but also to gain further knowledge on the behaviors of each individual participant.

We have more or less an idea of what teams we want, from the initial interviews and the CVs with cover letters that the students have submitted. But typically, we have some doubts in between putting this or that student together, or how certain students will behave when we put them together. So, we then go into the practical exercises, and we start moving students around in the different tasks to see if our assumptions were right.

Experiments were not only used to test possible groupings. Teachers also expressed during the interviews that prototypes could be used to test certain specific attributes from participants when put in difficult situations. Some examples of these tests were (1) groupings of individuals of the same background to eliminate skill advantage on solving the tasks and explore willingness to collaborate. (2) Students were grouped with others of similar personalities, such as multiple strong leaders in one group and more introverted students were placed in another, with the intention of revealing how their attitudes and behaviors were changed. (3) Teams that had already shown to work well, were given difficult or ambiguous tasks, and then examined on how they dealt with more adverse situations.

4.2.3. Experiential learning

The final theme that was found from the interviews with the teaching teams related to the evaluation, learning and modification of the complete process through experiential means rather than formal ones. During the final part of the interviews, teachers were asked several questions aimed to understand how they reflect, evaluate and improve their team formation process. From this line of questioning, it was found none of the teaching teams have implemented any intentional or formal reflective process to specifically converse about the process and possible changes and improvements. However, while recounting how their own process has evolved through the years, it was often commented that their team formation process had evolved through the years with the use of learned experiences and the connection of those experiences with theoretical findings. Yet often negative experiences and/or shortcomings from team/course performance played a more significant role in changing the team formation process to better respond to those shortcomings.

In this one specific case… a student, that had a very impressive CV and that dazzled all the teachers during the interview process, once the projects had started showed very conflictive behaviors and had big problems to work in a team and to collaborate with others. When that happened, we talked about how one thing is what people say during the interview and another one is how they behave in a real scenario. So, based on a previous experience that I had in team formation in another setting I suggested we implemented a [assessment exercise] to see the students in action.

Another learning mechanism that was commonly reflected on, had to do with learning from each other’s experiences. informal and “serendipitous” conversations with other teachers that might happen in the course of the projects drove decisions to either change or improve specific parts of their own process.

I felt more that when we traveled there was a nice, uhm, serendipity about conversations that I would have with [other teachers]. Things like, why did this person behave like this or acted like that? Or did we miss something as a teaching team?

Additionally, teachers reflected that as they gain experience from participating in team formation processes over many years, their observation skills are honed and changed by observations and decisions that they made in the past.

Sometimes, you start to see that the people are not behaving like you thought they would, things like a person who you thought were a quiet and shy person just takes charge of the team in bad moments. Other times, we have had [students] who presented themselves as very calm and easy-going, becoming very conflictive and you didn’t expect it at all, so it kind of blows up in your face. From cases like that you start to think of things that you may have missed [in the activities] and that helps you see them better in the future.

5. Discussion and conclusions

As any coach of a sports team knows, the combination of the best individual players does not necessarily make for the best team. Hence, approaches including assessments of team composition have the most potential. In that context, observer-assessment methods provide a deeper understanding of the students and their characteristics which leads to more informed team formation. However, they also present disadvantages to the team formation efforts. They require more time and heavily depend on the abilities of the observer(s) to properly recognize characteristics. And even then, there are unexpected team conflicts arising. Hence, the use of mixed methods that combine self-rating and observer-based methods may remain the preferred means of forming teams, as it allows the teaching teams to spot behaviors and skills with self-rating tools and then corroborate thought interactions between students for mismatches that could have been hidden by the participants in the self-rating measuring assignment. Furthermore, in cases where small trials through small assignments and trial team configurations allow teaching team members to not only observe said behaviors but also test them with different combinations of possible groupings of students.

Observer assessment does take time and may thus only be realistic in situations such as the ones described here, where teams are formed out of a relatively small group of (highly motivated) students. Experiences show, however, that they can be done in a meaningful way rather quickly, requiring a session of only a couple of hours. In light of the 9-month duration of the projects described here and the “cost” of conflicts arising later, they are definitely worth it.

Although the exclusive nature of this program, with smaller batches of students and ample teaching team time available, made SUGAR a sensible context for this study, it also has some limitations. The participating universities predominantly take industry-relevant programs, such as industrial design engineering, mechanical engineering and business programs. Furthermore, the students know that they are applying to a high-demand program and are usually highly motivated, reducing issues of free-riding by unmotivated students, as well as other issues that might come into play in regular large project-based courses. Finally, the amount of time teachers invest in these courses is usually higher than in regular education, thus perhaps making it hard to copy these practices in full in conventional education.

The choice for the SUGAR network as a case study, and within the network for a sample of the more experienced universities and teaching team members, seems justified as a context where best practices within design engineering education might be observed.

Our findings show however that current practices within the studied SUGAR network have evolved over time through trial and error. These evolutions and changes are partially grounded in literature from a multitude of fields related to the teaching teams’ background and not necessarily from team formation literature. This is then paired with experiential knowledge of important attributes and behaviors that are desirable for the execution of the projects to create team. However, despite a desire to enhance practices with further theoretically grounded methods or develop a more comprehensive process, limitations in time and resources often make it necessary to rely on the expertise and mindset of experienced practitioners. However, as far as post-project evaluations on the success or failure of formed teams have been done, they show only limited correlation to how well assessment practices aligned with team formation theory (unjustified success, unexplained fighting). Thus, providing directions for improvement both regarding theoretical understanding of student design team formation as well as for practical assessment sessions.