Emergency medicine residents’ beliefs about contributing to an online collaborative slideshow

Abstract

Objective

Collaborative writing applications (CWAs), such as the Google Docs^TM platform, can improve skill acquisition, knowledge retention, and collaboration in medical education. Using CWAs to support the training of residents offers many advantages, but stimulating them to contribute remains challenging. The purpose of this study was to identify emergency medicine (EM) residents’ beliefs about their intention to contribute summaries of landmark articles to a Google Docs^TM slideshow while studying for their Royal College of Physicians and Surgeons of Canada (RCPSC) certification exam.

Method

Using the Theory of Planned Behavior, the authors interviewed graduating RCPSC EM residents about contributing to a slideshow. Residents were asked about behavioral beliefs (advantages/disadvantages), normative beliefs (positive/negative referents), and control beliefs (barriers/facilitators). Two reviewers independently performed qualitative content analysis of interview transcripts to identify salient beliefs in relation to the defined behaviors.

Results

Of 150 eligible EM residents, 25 participated. The main reported advantage of contributing to the online slideshow was learning consolidation (n=15); the main reported disadvantage was information overload (n=3). The most frequently reported favorable referents were graduating EM residents writing the certification exam (n=16). Few participants (n=3) perceived any negative referents. The most frequently reported facilitator was peer-reviewed high-quality scientific information (n=9); and the most frequently reported barrier was time constraints (n=22).

Conclusion

Salient beliefs exist regarding EM residents’ intention to contribute content to an online collaborative writing project using a Google Docs^TM slideshow. Overall, participants perceived more advantages than disadvantages to contributing and believed that this initiative would receive wide support. However, participants reported several barriers that need to be addressed to increase contributions. Our intention is for the beliefs identified in this study to contribute to the design of a theory-based questionnaire to explore determinants of residents' intentions to contribute to an online collaborative writing project. This will help develop implementation strategies for increasing contributions to other CWAs in medical education.

Résumé

Objectif

Les applications d’écriture collective (AEC), telles que la plateforme Google Docs^TM, peuvent améliorer l’acquisition des compétences, la conservation du savoir et la collaboration en formation médicale. Ainsi, les AEC utilisées à l’appui de la formation donnée aux résidents offrent de nombreux avantages, mais la contribution au contenu relève du défi. L’étude décrite ici avait pour but de recueillir l’opinion des résidents en médecine d’urgence (MU) sur leur intention de présenter des résumés d’article novateur dans un diaporama Google Docs pendant qu’ils se préparaient à leur examen de certification du Collège royal des médecins et chirurgiens du Canada.

Méthode

Les auteurs, s’appuyant sur la théorie du comportement planifié, ont interrogé des résidents sortants en MU, en voie d’obtenir leur certificat du Collège royal sur leur intention de contribuer à un diaporama. Les premiers ont posé des questions aux seconds sur leur opinion concernant leurs croyances comportementales (avantages/inconvénients), leurs croyances normatives (groupes de référence favorables/défavorables) et leurs croyances de contrôle (obstacles/facteurs facilitants). Deux examinateurs ont procédé, chacun de leur côté, à une analyse qualitative du contenu à partir de la transcription des entrevues, afin d’en dégager les croyances saillantes en lien avec les comportements définis.

Résultats

Vingt-cinq résidents en MU sur une possibilité de 150 ont participé à l’étude. Le principal avantage mentionné de la contribution au diaporama en ligne était l’affermissement de l’apprentissage (n=15), tandis que le principal inconvénient était la surcharge d’information (n=3). Quant aux groupes de référence favorables, c’est celui des résidents sortants en MU, en voie de passer leur examen de certification du Collège royal qui a été mentionné le plus souvent (n=16). Peu de participants (n=3) ont fait état de groupes de référence défavorables. Le facteur facilitant mentionné le plus souvent était l’information scientifique de qualité évaluée par les pairs (n=9) et l’obstacle mentionné le plus souvent, les contraintes de temps (n=22).

Conclusions

Il se dégage de l’étude des croyances saillantes quant à l’intention des résidents en MU de contribuer au contenu d’un projet d’écriture collective en ligne, à l’aide d’un diaporama Google Docs^TM. Les répondants ont perçu, dans l’ensemble, plus d’avantages que d’inconvénients à la contribution au contenu et ils étaient d’avis que l’initiative recevrait un large appui. Toutefois, les participants ont relevé plusieurs obstacles qu’il faudrait aplanir pour accroître la contribution. Les auteurs ont l’intention, à partir des opinions exprimées dans l’étude décrite ici, de participer à l’élaboration d’un questionnaire fondé sur la théorie, qui permettrait d’examiner les déterminants de l’intention des résidents de contribuer à un projet d’écriture collective en ligne. L’exercice aidera à l’élaboration de stratégies de mise en œuvre visant à accroître la contribution des étudiants à d’autres AEC en formation médicale.

Background

The advent of the World Wide Web ¹ in 1991 led to an enormous range of innovations in medical education. ^{2 , 3} Now, over two decades later, the development of social media with its interactive content is expected to change the way medicine is taught. ^{4 – 8} “Social media” is defined as a group of network technologies that share a participatory approach for creating content through open architecture that facilitates collaboration. ⁹ Collaborative writing applications (CWAs) are a category of social media that allows multiple authors to contribute synchronously and asynchronously to a single document. ^{10 , 11} New CWAs, such as Google Docs^TM and wikis, are rapidly gaining popularity in medical education because they support free and open collaboration and decentralize content production. ^{4 , 7 , 9 , 12 – 18} Constructivist learning principles support the use of social media in medical education. Instead of passively receiving information, students can actively create course content, and by doing so increase their comprehension and knowledge retention. ¹⁹ Constructivism purports that students learn best when they construct their own meaning from experiences and develop their own solutions to problems. In the context of using CWAs in a collaborative writing project, the concept of individual constructivist learning is expanded to include communal constructivist learning, which engages students in developing their own information and creating knowledge that will benefit other students. ²⁰ In this model, students leave their own imprint in the development of their course, university, and ideally, discipline.

A meta-analysis assessing the efficacy of Internet-based learning in medical education found that such approaches have a positive effect. ²¹ However, the analysis did not include social media or CWA interventions. ²² Three recent systematic reviews found that social media improved skill acquisition, knowledge retention, student satisfaction, and teacher supervision. ^{11 , 23 , 24} A scoping review of the literature, ¹¹ focused specifically on CWAs, found three randomized trials ^{25 – 27} and concluded that CWAs had positive effects on scientific writing skills, leadership skills, and problem-based learning processes. However, low contribution rates have been found to be a major obstacle to CWA use in medical education. ^{11 , 23 , 28 , 29} This is a common problem with CWAs: for example, 44% of all contributions to Wikipedia (Wikimedia Foundation, USA) are made by 0.1% of editors. ^{30 , 31} This contribution process has been described as a “long tail” distribution, whereby a large number of people make small contributions. ^{32 – 34}

Since 2008, the authors have used Google Docs (now called Google Drive^TM) to share a collaborative online presentation designed for use by emergency medicine (EM) residents graduating from the Royal College of Physicians and Surgeons of Canada (RCPSC) training programs to allow them to share summaries and brief critical appraisals of landmark articles that they have reviewed in preparation for their RCPSC certification exam. The purpose of this presentation is to foster collaboration among EM residents across Canada and maintain an up-to-date database of article summaries. However, the contribution rate for article summaries has remained under 2%. ^{12 , 35} The purpose of this study was to identify residents’ beliefs about contributing summaries of landmark articles to an online national collaborative writing project using the Google Docs slideshow application, with the overall goal of designing strategies to improve contribution rates.

Methods

Theoretical basis

The Theory of Planned Behaviour (TPB) (Figure 1) ^{36 , 37} was used to identify residents' beliefs. This theory has been applied successfully to the study of health professional behaviours to help tailor implementation interventions. ^{38 , 39} A systematic review found that Internet-based implementation interventions informed by the TPB tend to have substantial effects on health behaviour change. ³ The TPB helps implementation scientists focus on key elements known to influence the adoption of clinical behaviours. Indeed, attitude, subjective norm, and perceived behavioural control are key TPB elements that must be considered when designing an effective implementation intervention. Together, these three factors explain a significant amount of behavioural change. ⁴⁰ When an individual has some control over a situation for modifying his or her clinical behaviour, intention is the immediate determinant of this behaviour. ⁴¹ In other words, before clinicians change behaviour, they must intend to do so. In turn, this behavioural intention is itself under the influence of a set of personal beliefs. According to Ajzen, an individual’s salient beliefs are the beliefs most frequently reported with respect to the key elements: attitude, social norm, and perceived behavioural control. “Attitude” refers to a person’s evaluation of the consequences (advantages and disadvantages) of adopting the behaviour. “Subjective norm” refers to a person’s perceived social pressure to engage or not engage in a given behaviour, including beliefs about how people who are considered important (referents) would like us to behave. “Perceived behavioural control” refers to a person’s perception of how easy or difficult it is to perform the behaviour in question in light of perceived barriers and facilitators (control beliefs).

Figure 1 Theory of planned behaviour

Study design

The protocol for this study has been published elsewhere ³⁵ and was approved by the Ethics Review Board at Centre de santé et services sociaux Alphonse-Desjardins.

Drawing on the TPB, we conducted semi-structured telephone interviews with graduating RCPSC EM residents. Interview questions were based on a vignette that clearly described the behaviour of interest that we wanted residents to reflect (Appendix 1, available at http://dx.doi.org/10.1017/cem.2014.49). The behaviour of interest was defined using Fishbein’s target-action-context-time (“TACT”) principles: ⁴² (1) target: a new Google Docs slide summarizing an important article missing from the presentation; (2) action: to contribute; (3) context: in preparation for the RCPSC certification exam in EM; and (4) timeline: within six months.

Study setting and recruitment of participants

Participants in the study had to be RCPSC EM residents in their last year of training. Thirteen of the 14 RCPSC EM programs in Canada had eligible residents at the time of the study. Potential participants were recruited via the annual National Emergency Medicine Review course at Queen’s University. During the course, the principal investigator presented a 90-minute lecture reviewing the most important literature of the year. A similar lecture has been given every year since 2008 using the Google Docs slideshow application. After the lectures in 2010, 2011, and 2012, an email was sent to all attendees inviting them to freely access, edit, and update the slideshow based on their own studies and readings. The email included an invitation to participate in a telephone interview, the vignette, and the link to the study’s SurveyMonkey^TM questionnaire. Email reminders were sent one week and two weeks after the initial email. In addition, invitations were sent to specific participants to obtain a representative sample of residents from each training program, and to elicit the widest possible range of beliefs. In order to identify these additional participants, we contacted local leaders from the universities where the email invitation response was low and asked them to identify potential participants. Potential participants identified through this approach were sent a personal email invitation to participate.

Data collection procedure

Consent was obtained before all interviews. Using a SurveyMonkey questionnaire (see Appendix 2, available online at http://dx.doi.org/10.1017/cem.2014.49), information was collected on participant’s age, gender, degree, university affiliation, previous consultation and edition of any Google Docs application or other CWA (e.g., Wikipedia) used. The questionnaire asked if participants had consulted or edited the online slideshow, and if so, how often and what changes they had made (e.g., adding, correcting). Participants were asked if they knew of articles that were missing, if they preferred another CWA, and if they had downloaded the slideshow for personal use. Using a 7-point Likert scale, participants were asked to rate their intention to contribute a new slide summarizing an important missing EM article within the next six months. A few days after the online questionnaire was completed, an interviewer, blinded to the online responses, conducted a telephone interview with each participant. The interviewer read the vignette to each participant and elicited their beliefs regarding the targeted behaviour. Each interview was digitally recorded and anonymously transcribed verbatim. Interviews were conducted in French or English.

Sample size

Godin and Kok ³⁸ suggest that a sample size of 25 participants is sufficient to achieve data saturation when conducting a salient beliefs study. To verify that data saturation was reached with our sample, we followed principles described by Francis and colleagues ⁴³ and used a cumulative frequency graph to identify the data saturation point (i.e., three consecutive interviews without any new reported beliefs after the initial 10 interviews) for each of our three constructs (behavioural, normative, and control beliefs).

Data analysis

For participants’ demographic characteristics and their self-reported contributions to the online presentation, the mean, standard deviation, median, and interquartile range for continuous variables was calculated, as was the percentages for dichotomous variables. Because of limited information about the population of all eligible residents, the study was only able to assess whether the sample differed significantly from all eligible residents in gender distribution and distribution of EM training programs. All calculations, including the χ² test to calculate p-values for categorical data, were performed with Microsoft® Excel for Macintosh 2011 (Microsoft, Redmond, WA, USA).

For qualitative data, a mix of inductive and deductive thematic analysis was used. Two authors (SG, JT) independently analysed transcripts of the recorded interviews to identify control, normative, and behavioural beliefs. Through discussion, grouped beliefs were inductively placed into themes to produce a list for each construct. In order to compare the study results to those of similar studies, labels from a validated taxonomy of social media adoption determinants were used. ^{11 , 44} Each belief was ranked from most to least frequently mentioned. The top 75th percentile of most frequently cited beliefs were considered “salient” in keeping with TPB methodology. ⁴⁵ In cases of disagreement, a third author (PA) was consulted.

Results

Data saturation

Data saturation was subjectively achieved during the interview with the 17th participant. However, we continued recruitment until our intended sample size of 25 participants was reached. Using the methods proposed by Francis and colleagues, ⁴³ we retrospectively determined that data saturation had been attained for control, behavioural, and normative beliefs after the 14th, the 13th and the 16th interviews, respectively (Figure 2).

Figure 2 Cumulative frequency of reported beliefs by Canadian emergency medicine residents for the studied behavior between 2010–2012 to determine the data saturation point.

Participant characteristics

Demographic characteristics of the eligible and participating study population are presented in Table 1. Over the three years of this study, a total of 150 residents were eligible to participate (37 in 2010, 49 in 2011, and 64 in 2012). Of these, 77% (n=115) did not respond to our invitation, 4% (n=6) declined, and 3% (n=4) of email addresses were invalid. A total of 17% (n=25) of participants were recruited. The participation rates for each respective cohort were 11% (n=4), 20% (n=10), and 17% (n=11). Demographic information was collected for all participants except one, whose data were lost. Residents were recruited from 10 of the 13 EM programs. Among participants, 38% (n=9) had previously edited another Google Docs document and 8% (n=2) had previously edited a wiki.

Table 1 Demographic characteristics of eligible and participating study population

* Demographic information is missing for one participant because his/her responses were not recorded in the online survey

Abbreviations: n: number; SD: standard deviation; 25%–75% IQR: 25th to 75th percentile interquartile range; University U1–13: to preserve the confidentiality of each participating university, we named the 13 different universities U1 to U13.

SurveyMonkey^TM on contributions to Google Docs^TM

Among the 24 participants with SurveyMonkey^TM data, only 17% (n=4) stated that they had edited the online presentation (Table 2). Although 50% (n=12) of participants had read an article in the past year that they considered should be included in the slideshow, only 8% (n=2) added a new summary to the presentation. One participant had downloaded the slideshow and made offline edits. A higher proportion of participants (75% [n=18]) reported having consulted the slideshow in the last year. Four participants said they would prefer a different collaborative platform (e.g., Dropbox^TM or a blog). Finally, although the self-reported contribution rate was low, 29% (n=7) of participants stated that they were “quite likely” or “very likely” to contribute a slide to the slideshow within the next six months.

Table 2 Participants self-reported use of the study Google Docs online slideshow

* One participant’s responses were not recorded in the online survey and thus were lost. This participant's responses to the semi-structured interviews are included.

Residents’ salient beliefs

After interviewing all 25 participants, 14 behavioural, 12 normative, and 30 control beliefs were identified (Table 3). Of all these, seven behavioral, seven normative, and 16 control beliefs were considered salient.

Table 3 Reported beliefs of study participants regarding contributing to an online Google Docs slideshow

* Beliefs identified with asterisk were considered salient because they represent the top 75 percentile most frequently cited beliefs.

^§ The rank number corresponds to the position held in the ranking of all beliefs. The most frequently mentioned belief is ranked first. The ranking numbers do not necessarily follow each other in this table, since we grouped them as advantages, disadvantages, favorable referents, unfavorable referents, barriers, and facilitators. Two beliefs can hold the same rank when they were mentioned at the same frequency by our participants.

^† n=the number of participants who reported the belief during their interview, and %=the number of times the belief was reported in all interviews divided by the number of times all beliefs in that category (behavioral, normative, and control beliefs) were reported in all interviews.

^‡ The label for this belief was taken from the Gagnon and colleagues <44> framework.

^¶ The label for this belief was taken from the Archambault and colleagues <56> framework.)

^∆ ICT: Information and communication technologies

Behavioural beliefs

Eleven advantages and three disadvantages were reported by the participants. Interestingly, 76% (n=19) of participants saw no disadvantages to contributing to the slideshow. The three behavioural beliefs mentioned most frequently were that contributing a slide would “consolidate learning” (15% [n=15]), would be “useful” (14% [n=14]), and would allow “mutual aid and collaboration” (13% [n=13]), where the denominator is the total number of beliefs. All salient behavioural beliefs were about advantages. Some non-salient beliefs about disadvantages were reported, such as the belief that contributing to the slideshow added to “information overload” (n=3).

Normative beliefs

The three most frequently mentioned positive referents were “graduating EM residents writing the exam” (26% [n=16]), “all EM residents” (21% [n=13]), and “EM teachers” (13% [n=8]). All salient normative beliefs concerned positive referents; 88% of participants (n=22) did not identify any negative referent. Only two disapproving referents were identified: “competitive people” reluctant to share information with others (3% [n=2]), and regulatory authorities who might disapprove of sharing information (2% [n=1]).

Control beliefs

Sixteen control beliefs were considered salient: eight facilitators and eight barriers. The top three were barriers: “time consuming” (13% [n=22]), “other priorities in the last year” (9% [n=15]), and “online presentation not easy to access” (8% [n=14]). The top three facilitators were “peer-reviewed high quality scientific information” (6% [n=10]) provided by the slideshow, “the use of a template” (5% [n=8]), and “familiarity with information and communication technologies” (5% [n=8]).

Discussion

Summary of findings

This study identifies EM residents’ beliefs about contributing to an online collaborative slideshow via Google Docs applications designed to help them share summaries and critical appraisals of landmark articles in preparation for the RCPSC EM certification exam. The study used TPB methodology to identify the salient beliefs necessary for designing a theory-based intervention to increase contributions to collaborative writing projects in medical education. Overall, participants perceived more advantages than disadvantages to contributing and believed that this initiative would receive wide support. However, participants reported several barriers that need to be addressed to increase contributions.

Among the major themes explaining the low contribution rate were organizational factors (e.g., lack of time, other priorities in the last year of residency), technical factors related to the CWA (e.g., low accessibility, task complexity, lack of usefulness), and individual factors (e.g., fear of making a mistake, lack of motivation, poor computer literacy).

Clinical relevance

These findings are important because CWAs are increasingly used in medical schools and residency programs worldwide. ^{18 , 25 , 26 , 29 , 46 – 54} Beyond helping learners pass an exam, CWAs have the potential to keep students updated on relevant evidence, train them to quickly summarize it, and teach them to collaborate and share information in time-sensitive clinical contexts (e.g., EM, critical care). These study results are consistent with previous findings indicating that while the intention to contribute is high, contribution rates to collaborative projects remain low. ^{11 , 23 , 28} As highlighted in recent systematic reviews, if these applications are expected to produce convincing and positive results in teaching medical skills and knowledge, their implementation must be optimized. ^{11 , 23 , 24}

The use of CWAs in medical education has been explored using a variety of theory-based approaches. However, few authors have used this approach to understand how learners’ or clinicians’ beliefs influence their likelihood to contribute to collaborative online projects. While McGowan and colleagues used a quantitative approach to study the determinants of the “use of social media applications to share medical knowledge with other physicians,” ⁵⁵ this study used TPB-based qualitative methods to generate an exhaustive list of all the salient beliefs that influence contributions to a collaborative writing project. This list could be useful in creating questionnaires to investigate other collaborative writing projects and designing interventions aimed at increasing contributions. ⁵⁶

Time constraints were found to be a major barrier. This is common to all collaborative writing projects, especially among health care professionals and students. ^{27 , 57 – 61} Although there is no simple solution, some authors have proposed incentives such as offering continuing medical education or residency training credits for contributions. ^{62 , 63} A new set of scholarly impact metrics that could measure the impact of open-source contributions might also provide further motivation. ⁶² Study participants suggested offering prizes for contributions and making the slideshow available earlier in their residency. Finally, convincing participants of the usefulness of contributing to the CWA is another potentially important solution. ⁵⁵ The negative beliefs identified (barriers, negative referents, and perceived disadvantages) are similar to those reported in a recent systematic review: information overload, perceived unequal distribution of work, competitiveness, regulatory authorities preventing access, time constraints, lack of motivation, and lack of computer literacy. ¹¹ Difficulty accessing the online presentation was a common barrier; 33% of study participants had downloaded the slideshow to their personal files. The only unique barrier identified, although not salient, was the lack of a track changes feature in the Google Docs slideshow platform. Four participants (17%) suggested using other collaborative tools (e.g., Dropbox or a blog) to share summaries, although it should be noted that these do not allow for better tracking of changes and have other disadvantages. ⁶⁴ It may be helpful to assess other relevant platforms, such as Google Sites^TM, to address these issues.

As reported previously, the top three facilitators identified were the quality of information, the provision of a template for contributions, and training for CWA use. ¹¹ While the number of perceived advantages and positive referents was notably higher than the disadvantages and negative referents, the top three control beliefs were barriers, and overall, more barriers were identified than facilitators. Future studies measuring the importance of each of these barriers would be necessary to identify which contribute the most to low contribution rates. Interventions to increase contribution rates could then better target these barriers.

Strengths

To the authors’ knowledge, this study is the first to use a qualitative theory-based approach to identify residents’ salient beliefs concerning their intention to contribute to an online collaborative writing project. These study results provide baseline data for comparison with future studies. Furthermore, the detailed and rigorous description of qualitative content analysis provided in this study will enable other researchers to reproduce this study’s approach. The planned sample size (n=25) was based on Godin and Kok’s work, ³⁸ even though the subjective data saturation was reached at the 17th interview. This allowed verification of the principles of data saturation proposed by Francis and colleagues. ⁴³ This method could help future qualitative researchers operationalize data saturation determination and avoid unnecessary patient recruitment and resulting resource usage. Finally, this study explored three distinct cohorts of participating residents over the course of three years, thus allowing the presentation of data on a broader and more generalizable description of salient beliefs.

Limitations

This study had limitations that should be considered. First, we did not perform member checking. However, two independent research professionals experienced in TPB methodology carefully analysed the transcripts and resolved disagreements through a rigorous approach. The inclusion of verbatim quotes for each salient belief was intended to enable readers to judge the interpretation of the results. Second, our sample size of 25 participants represents only 17% of all eligible EM residents and failed to include all universities, potentially limiting the generalizability of the results. Moreover, the results may be affected by a positive response bias (with one university, representing 5% of the sample, providing 25% of the responses) and a negative non-response bias (with 25% of the sample providing 0% of the responses). The participating residents could have been more likely to have used and/or felt favorably about the product than those that did not. Since we did reach data saturation, we feel it is unlikely we would have identified new beliefs even if more residents and all universities had been represented. Among the elicited salient control beliefs, the top three were barriers indicating that the current participants shared many negative views about this project. Third, the measurement of future intention to contribute a slide to the collaborative writing project is only a proxy of the actual behaviour and may also be influenced by a “social desirability” bias, with participants responding with what they believe the authors wanted to hear. Alternatively, the survey participants may represent a small group of active contributors that represent the “long-tail” distribution described in other collaborative writing projects. ^{32 , 33} Since the Google Docs slideshow application did not allow precise measurement of how many new slides were added, the study could not verify whether the measured intention to contribute resulted in the completion of the actual behaviour. Future studies could utilize web analytics to investigate this relationship. Finally, the studied behavior was complex and some participants may not have understood the difference between “contributing to” and “using” the slideshow, although efforts were made to minimize this problem (e.g., presenting a clinical vignette, conducting interviews after participants had the opportunity to contribute).

Conclusion

EM residents share different salient beliefs regarding their intention to contribute content to an online collaborative writing project using a Google Docs slideshow. Overall, participants perceived more advantages than disadvantages to contributing and believed that this initiative would receive wide support. However, participants reported several barriers that need to be addressed to increase contributions. The researchers’ intention in the next phase of this initiative will be to construct a questionnaire to measure the quantitative importance of the salient beliefs identified in this study and to conduct another survey with a larger and more representative sample. This will allow the researchers to prioritize the most important beliefs that should be targeted by a theory-based intervention aiming to increase contributions to the collaborative writing project and will help develop implementation strategies for increasing contributions to other CWAs in medical education.