Mentoring up: How mentorship training influences mentee intent to change behavior

Abstract

Introduction:

The purpose of this study was to evaluate data from different implementations of the Mentoring Up curriculum, designed by the Center for the Improvement of Mentored Experiences in Research. The study investigated the relationship between participants’ self-reported change in mentoring competence and the behaviors they intended to implement post-training.

Methods:

The data set included 401 respondents who consented to participate after 59 Mentoring Up training events hosted by 34 institutions between 2015 and 2022. Responses to the Mentoring Competency Assessment (MCA) were analyzed to determine which factors were related to self-reported changes in participants’ mentoring competencies post-training.

Results:

Quantitative analysis showed that intent to change, perceived value of training, training modality, and prior mentor training were all significantly associated with the magnitude of change in MCA scores between pre- and post-tests. Further, participants who engaged in face-to-face training found significantly more value in the training than those who participated online. Analysis of open-ended questions demonstrated that participants with larger changes in MCA scores were more likely to address core principles of Mentoring Up curriculum when discussing their behavior change plans post-training.

Conclusion:

Participants improved their mentoring competence by participating in the Mentoring Up curriculum, and this change was significantly and practically associated with an intent to modify their behavior in their mentoring relationships.

Introduction

Mentorship in STEMM (science, technology, engineering, mathematics, and medicine) is understood as a “professional working alliance in which individuals work overtime to support the personal and professional growth, development, and success of the relational partners through the provision of career and psychosocial support” [1]. In higher education, research across disciplines has universally demonstrated many positive outcomes for those who report having an effective mentor [2–5]. Given these consistent findings, a variety of mentorship training approaches have emerged for mentors to become more effective [6,7]. Traditionally, most research and training have focused on improving the ability of the mentor to provide career and psychosocial support to the mentee [8,9]. One such program, Entering Mentoring, was designed by Handelsman, Pfund, and colleagues, to improve the mentoring skills of participants and has been adapted for use across career stages. Entering Mentoring uses a process-based approach to training where mentors and mentees collaborate and discuss mentorship across a range of different contexts [6]. The research that supports this training identified six critical skills for effective mentoring, including maintaining effective communication, aligning expectations, assessing understanding, addressing equity and inclusion, fostering independence, and promoting professional development [6].

The efficacy of the Entering Mentoring training programs has been well-researched. The Mentoring Competency Assessment (MCA) [10] was developed as a questionnaire to measure changes across core competencies in Entering Mentoring. This instrument has repeatedly demonstrated acceptable psychometric properties and has recently been revalidated [11]. The MCA instrument consists of 26 items on a Likert-type scale (1 = not at all skilled, 4 = moderately skilled, 7 = extremely skilled) and rates mentors’ skills across the competencies that build the core curricula of Entering Mentoring, such as effective communication and aligning expectations. Data collected using this instrument have shown that participants across career stages who participated in the Entering Mentoring curriculum perceived a significant positive change in their mentorship skills post-training [6].

However, given the reciprocal nature of mentoring relationships, a growing emphasis has been placed on likewise empowering mentees with the skills required for successfully navigating mentorship relationships. Entering Research, for example, is a process-based training program designed for novice undergraduate researchers to navigate challenges with research mentors [12]. The students in these programs reported significant improvement in their ability to communicate and interact with their research mentors [13]. For mentees further along their research career, “mentoring up” is a concept derived from “managing up” which proposes a proactive engagement of the mentee to improve the overall mentoring relationship [14,15]. Lee et al. theorized [15] that an integrated approach bringing together two evidence-based mentoring programs (Entering Mentoring and Entering Research) demonstrated that the core principles upon which they were based aligned conceptually with managing up. It is from this foundation that the Mentoring Up training was developed. The curriculum, which is available to both mentees and early-career mentors, is a way to “empower mentees to be active participants in their mentoring relationships by shifting the primary emphasis from the mentors’ responsibilities in the mentor-mentee relationship to acknowledging the mentees’ contributions of equivalent importance” [16]. To date, only one study has specifically addressed the effectiveness of Mentoring Up training on early-career researchers such as graduate students and postdoctoral fellows [16]. The purpose of this article therefore is to report more in depth on the evaluation data from different implementations of the Mentoring Up curriculum. Specifically, this study sought to determine which factors (demographics, title, career, stage, prior mentorship training, intent to change behaviors, training dosage and modality, and perceived value of training) were related to self-reported changes in participants’ mentoring competencies following training. Given the importance of the mentoring competencies as a vehicle for individuals’ changes in behaviors in their mentoring relationships in the past Entering Mentoring trainings [6,16,17], this study also sought to determine the relationship between participants’ self-reported change in mentoring competencies to the strategies they report implementing post-training.

Methods

Mentoring Up training survey data were collected nationally from 2015 to 2022. The survey was completed by 911 mentees after 70 Mentoring Up training events hosted by 38 institutions. In this study, 11 training events either implemented a single module for beta testing or failed to collect participant information and were excluded from the analysis. The final dataset therefore included 401 consenting respondents from 59 Mentoring Up training events hosted at 34 institutions. These participants engaged in training either in person or online and completed the MCA in surveys following training. Table 1 shows the sample characteristics of the respondents. Of the 401 participants, 61% identified as women, while 35.8% identified as men. The participants were predominantly White (55.5%), while 12.8% also reported Hispanic or Latino ethnicity. The respondents self-identified 13 different professional titles, but most were graduate or professional students (40.4%), postdoctoral fellows (30.3%), assistant professors (19.9%), or researchers/scientists (9.4%). The training was delivered through two modalities, online and in person, with 66.5% participating in person. In addition, 22.3% of the participants indicated prior mentorship training of some kind. The average participant engaged in 5.314 hours of training. Based on previous studies, four or more hours of training had been identified as the smallest effective mentorship training dosage, so participants were categorized into low-dosage (<4 hours) (46.2%) and high-dosage (4+ hours) (53.8%) groups [18].

Table 1. Characteristics of respondents

Demographic characteristics are nonexclusive and self-selected.

The 26-item MCA, as adapted by Sancheznieto for mentee participants [16], served as the primary outcome for the Mentoring Up curriculum and asked mentees to self-assess using a retrospective pre-post-survey for their own skills in mentoring at the conclusion of the training. One participant was excluded from the analysis for not completing post-test responses. The remaining 400 participants were grouped into four equivalent quartiles based on the degree of change between their pretest and post-test responses.

The participant’s scores were ranked ranging from the lowest to the highest change scores. Based on these rankings, participants were categorized as either high change, moderate-high change, moderate-low change, or low change. Chi-square tests of independence and qualitative analysis of open-ended survey questions were used to measure intent to change and degree of change after participating in facilitated Mentoring Up training. The Chi-square tests analyzed nine grouping variables, including intent to change, training dosage, modality, prior mentor training, the value of the training, gender, race, ethnicity, and professional title. In these analyses, a standardized residual model described by Agresti [19] was employed because the Chi-square test is an omnibus statistic that fails to specify which categories contribute to statistical significance [20]. The significance level for analyzing the adjusted residuals was modified by the Sidak [21] method (1-(1-α)1/t where t = number of tests run) to control for Type I error rate. With the largest number of adjusted residual comparisons being 12, the significance level for the standardized residuals was α = .0042. Given that, adjusted standard residuals greater than an absolute value of 2.87 met the requirements for significance at p = 0.0042 [19,22]. Quantitative data were analyzed using the IBM SPSS statistics 27.

Additionally, qualitative data analysis was conducted on one open-ended question that was included in the post-test which asked participants to describe behaviors they intended to change after the training concluded. Participant responses were grouped and analyzed based on their change category. The qualitative analysis used an open coding process, described by Williams and Moser [23], to categorize open-ended responses. This included an organization of participant responses using similar words and phrases and concept indicators into thematic domains. One member of the research team analyzed all participant responses 3 times iteratively and refined the initial themes into 16 codes. It is worth noting that the codes identified in this qualitative analysis aligned with codes developed in a parallel research project [16], which conducted a different, independent analysis on parts of the same dataset. This demonstrates the content validity of the codes generated.

Results

Quantitative analysis

Chi-square tests of independence were run to determine if there were significant associations between key variables and change categories. The full results are listed in Table 2. The first Chi-square test revealed a significant association between MCA change category and likelihood to make or plan to make a change in mentorship behavior, χ ² (3, N = 394) = 73.561, p < 0.001, Cramer’s V (ϕ_c) = .432. The low change group was significantly more likely to report no change than any other quartile, and the effect size statistic indicated a relatively strong association [24,25]. There was also a significant correlation between overall MCA change scores and the likelihood that one would recommend the training (measured on a 1–5 scale) to a friend or colleague (r = 0.358, p < 0.001, data not shown).

Table 2. Chi-square analysis of key variables by MCA change category

Note: Student includes undergraduate, graduate, and professional; MCA = Mentoring Competency Assessment; F2F = face-to-face; PD = postdoctoral; Clin = clinical.

*p < .005; ** p < 0.001, residuals significant > |2.87|.

The second Chi-square test was conducted to determine the relationship between training dosage and MCA change category. The Chi-square test revealed that there was no significant association between MCA change category and training dosage, χ ² (3, N = 400) = 5.602, p = 0.133, ϕ_c = .118. Further, there was no significant correlation between total hours in training and overall change scores (r = 0.015, p = .764).

The third Chi-square test sought to show the relationship between training modality and MCA change category. The Chi-square test revealed that there was a significant association between MCA change category and training modality, χ ² (3, N = 400) = 15.127, p = 0.002, ϕ_c = .194. Participants in the face-to-face training were more likely to report change scores in the high change category, whereas students in the online training were more likely to report change scores in the low change category, with effect size demonstrating a weak association between the variables.

The fourth Chi-square test was run to determine the relationship between MCA change category and participants having participated in prior mentorship training. The Chi-square test showed that there was a significant association between MCA change category and prior mentor training, χ ² (3, N = 272) = 12.667, p = 0.005, ϕ_c = .216. Participants with prior mentor training were more likely to report change scores in the low change category whereas participants with no prior mentor training were more likely to report MCA change scores in the high change category. Effect size data indicated a moderate association between these variables.

The fifth Chi-square test was conducted to determine the association between MCA change category and whether the training was viewed as valuable (yes or no). The Chi-square test showed significant associations between MCA change category and whether participants viewed the training as valuable, χ ² (3, 395) = 90.320, p < 0.001, ϕ_c = .478. Respondents within the low change category were more likely to report no value in the training, and the effect size statistic indicated a relatively strong association. Further, follow-up Chi-square tests demonstrated that modality was significantly associated with seeing value in the course. Face-to-face students were more likely than online participants to rate the course as valuable, χ ² (3, 395) = 16.197, p < 0.001, ϕ_c = 0.202 (data not shown).

The sixth Chi-square test was run to demonstrate the relationship between gender and MCA change categories. The Chi-square test showed that there was no significant association between gender and MCA change score quartiles, χ ² (3, 400) = 3.447, p = 0.328, ϕ_c = .098.

The seventh and eighth Chi-square tests sought to show the relationship between race, ethnicity, and MCA change category. The Chi-square tests showed that there was no significant association between race, χ ² (3, 400) = 1.66, p = 0.646, ϕ_c = 0.064 or ethnicity χ ² (3, 386) = 2.68, p = 0.444, ϕ_c = 0.083 and MCA change score quartiles.

The ninth Chi-square test was conducted to determine the association between MCA Change category and self-reported professional title. The participants identified 13 different titles including 2 assistant deans, 4 associate professors, 73 assistant professors, 31 researchers/scientists, 5 associate researchers/scientists, 6 assistant researchers/scientists, 11 lecturers/instructors, 123 postdoctoral fellows, 7 clinical fellows, 159 graduate students, 10 undergraduate students, 8 medical students, and 11 other. Many participants selected multiple titles. In an effort to analyze the title, three primary categories were created, which included students (all grad, professional, and undergraduate), postdoctoral and clinical fellows, and professors/lecturers (which included all professors and lecturers). The researcher’s title was frequently chosen with one of these three major categories. Participants who selected two or more of the major categories were excluded from the analysis. The Chi-square test showed that there was no significant association between professional title and MCA change score quartiles, χ ² (3, 366) = 12.82, p = 0.046, ϕ_c = .187.

Qualitative analysis

Sixteen codes were identified through open coding of participant responses to the one open-ended question, “Have you made, or do you plan to make, any changes in your own practices as a mentee or in your relationship with your mentor(s) as a result of this training?” The codes were grouped by MCA change score category to determine if there were any patterns in participant responses based on the magnitude of their likelihood to change mentorship behavior (see Table 3).

Table 3. Participant codes grouped by MCA change category based on responses to the prompt have you made, or do you plan to make any changes in your own practices as a mentee or in your relationship with your mentor(s) as a result of this training?

MCA = Mentoring Competency Assessment; CIMER = Center for the Improvement of Mentored Experiences in Research.

Respondents in the low MCA change category were far more likely to indicate no change or to not comment than any other category. Of the 100 respondents, 49 did not respond, and 10 indicated no changes would be made. These respondents used 1027 words, by far the fewest of any category to report their change details. Of these participants, 41 reported a change score of 0 or less, with 6 reporting negative change scores. Participants in this group were by far the most likely to not respond or to report the training was unhelpful. For example, one participant said, “This training has convinced me that diversity training is counterproductive. You need to focus on the individual, and discard programs that lump people by race, sex, or other attributes.” For participants with the highest change scores in this category, responses demonstrated a lack of hope that mentoring up could influence their mentor to change, for example, one respondent reported, “None (change), I’’m not confident my PIs will be open to using these strategies in the future.” Of those who did report a change, the most frequent change reported was adopting mentor compacts or behavioral contracts and using materials from training to guide their mentor relationships. One respondent indicated, “I plan to utilize a mentoring map. I also will form, in writing, my mentoring philosophies for undergraduates, graduates, and postdocs that I will use in my future lab.”

Respondents in the moderate-low MCA change category were far more likely to respond to the change prompt than the low MCA change category. Only 21 of the 100 respondents failed to respond, and only 1 participant explicitly indicated no plans for change. These respondents used 1593 words to describe their change plans. Participants in this category were most interested in improving their communication ability with mentors and mentees. Forty respondents reported changes in either maintaining effective communication or being more direct, assertive, and proactive, which was more than any other MCA change category. One example of a respondent indicating a change in communication method with their mentor said, “I plan to communicate with my mentor in much more straightforward ways as a result of this training experience.” When considering their approach to mentees, one respondent indicated, “I will be deliberate about mentorship. I will clearly communicate expectations and will foster a bidirectional relationship with future and current mentees.”

Respondents in the moderate-high MCA change category failed to comment 23 times, with 1 respondent reporting no change. However, the respondents in this category reported no change necessary because they felt their relationship with their mentor was already excellent but did indicate they would consider the training the future when they became a mentor.

No (change), but the only reason I say that is because the seminars confirmed just how good my relationship is with my mentor. There were many useful themes and strategies that came up that I will internalize and use when I am assuming the role of mentor.

Respondents in this category used the most words, 1847, to indicate their change plans. Participants in this category were also interested in improving communication but were more likely than previous categories to report a desire to align expectations and work on goal setting, a desire to be more organized and to set more meetings, and a wish to find more mentors for their mentoring network. One participant reported that they would,

More explicitly talk about goals and expectations and write them down together. Work hard to build trust and confidence by sharing more of the research process and challenges with my mentee and keeping abreast of her own school and professional goals.

From the mentee perspective, one participant indicated that,

I have been more proactive in engaging and meeting regularly with my mentor. I have also begun to take charge of my own career and development and have been employing “managing up” techniques to effectively get what I need to help me in this endeavor from my mentor. In the future, I plan to discuss expectations and communication styles at the beginning of a mentor-mentee relationship to make sure that they are aligned.

Participants who had been both mentors and mentees also felt that more meetings would be helpful. For example, one mentee suggested, “I plan to schedule a 1 on 1 meeting with my advisor once a month instead of once a semester to make sure we have a stronger relationship and that our expectations are met.” One mentor indicated simply, “Regular meeting with mentees.” Although some respondents did indicate needing a new primary mentor, most were interested in expanding their mentor network. For example, one respondent reported, “Try to set up regular weekly or bi-weekly meetings with my advisor, work on building a support/mentoring network beyond my primary advisor.”

Respondents in the high MCA change category were the most likely to comment. No participants in this category indicated any change. The 81 participants who responded did so with 1769 words and were more likely than any other group to report a change in aligning expectations, maintaining effective communication, building more self-efficacy, reflecting on biases, and reevaluating their perceptions of their mentoring relationships. One participant suggested that they would work to improve expectations both for mentors and mentees, “I have used some of the aligning expectation documents to better prepare me for helping my mentees, as well as the mentoring I am receiving.” A mentee reported that they would “Be clearer in communication. For my next postdoc, I will do a much better job aligning expectations.” Improving communication and setting expectations were often mentioned together, with nine responses being coded for both. Self-efficacy was not often mentioned by respondents in this study but by far the most frequently in the high MCA change category. One respondent said they wanted to:

Spend quality time doing my research, enjoy my research. Work hard on getting the stage of being more self-efficacy in association with an adequate balance of delegation of work, meaning by this, promote the team work rather the isolate pattern (sic).

Although there were several comments throughout the study from respondents regarding reevaluating mentorship and mentoring relationships, these comments were most reported in the high MCA change category. For example:

I’ve been thinking more about my agency with regard to my relationship with my mentor. Before the training, I was really frustrated with our relationship but didn’t feel like there was anything I could do about it. Now I feel more empowered and have ideas about how I might approach that first conversation, but I’m still mustering the will to contend with the power dynamics. I guess you could say that before the training, I was in pre-contemplation, now I’m in a contemplation phase.

Participants in this category were also most likely to discuss what they learned about bias from the training. One participant indicated, “I will clarify expectations and practice more active listening. How inclusion has been interwoven into mentoring relationships was another important take-away for me.”

Discussion

The results of the training evaluation suggest that the Mentoring Up curriculum was successful in changing perceptions of mentoring competence in early-career researchers. Of the 400 participants, 314 reported that they intended to change their behavior post-training to improve the mentoring relationships. In addition, only six participants self-reported no change or negative scores on the MCA post-training. The results also showed that participants were more likely to report greater intent to change behavior when they perceived greater changes in mentoring competence. Those participants who reported the greatest change in their MCA scores were also more likely to specifically address core principles in the Mentoring Up curriculum, such as maintaining effective communication or aligning expectations, when discussing their behavior change plans post-training, suggesting that the training was highly valuable to those individuals. This is further supported by the significant association between participant perceptions of the value of training and the magnitude of change in their self-evaluations of mentoring competence. Of note, mentees suggesting that they would change their behavior by being more proactive was not the most highly reported behavior change. If creating more proactive mentees is the goal of Mentoring Up training, more emphasis may need to be placed on promoting assertiveness and proactive mentee behavior in the future.

Connected to this issue of value, participants with prior mentor training were far more likely than their peers to perceive less change in their mentoring competence from participating in the Mentoring Up curriculum. Research has demonstrated prior mentor training generally reduces the impact of Entering Mentoring training gains on participants, so this finding was not particularly surprising [26]. Despite the smaller magnitude of change, several respondents with prior mentoring training did report that they would adopt materials from the training (like the Individual Development Plan) that they liked or that they would be more organized in their mentoring relationships. Although these changes may not mirror large changes in perceptions of mentoring competence, they do represent important positive changes as a result of the training that participants may not traditionally associate with the core competencies measured by the MCA.

One noteworthy finding in this study is that training modality was significantly associated with perceived change in mentoring competency. Participants in the face-to-face curriculum were much more likely to report a greater magnitude of change than online attendees. This contradicts previous findings [26] from researcher mentor training using a similar curriculum (with an older, more experienced sample) that found modality had no significant impact on participant skill gains. Participants in the face-to-face modality were also significantly more likely to report seeing more value in the course. This is an important finding, especially given the proliferation of online curriculum in the post-pandemic era and the fact that most post-training surveys have found no differences in trainee satisfaction between online and face-to-face modalities [27,28]. This potentially suggests that online training may not have as much value for younger, less experienced groups. However, further studies that directly compare different modes of implementation across experience levels are warranted, given the different analytical methods employed by our two studies. Because intentional and well-designed online training offerings might increase accessibility and dissemination of training, it is critical to consider whether or not they are as effective as in-person implementations and how to improve them if they are not. Finally, gender, race, ethnicity, and title (which could be viewed as a replacement for professional experience) showed no significant association with perceived change in mentoring competence.

Limitations

There are several limitations to report in this study. First, the data collected were self-reported, cross-sectional survey data, and therefore, causation cannot be determined. Second, the data in this study were converted from continuous to categorical in order to use nonparametric statistical analysis. Altman and Royston [29] suggest that changing continuous data to categorical variables can reduce statistical power and cause information to be lost. However, much of the focus of the statistical community [24,30,31] is on issues with dichotomization. Altman and Royston [29] indicate that if a careful analysis of the data suggests that converting to categories is appropriate, it is best to do so with more groups in order to control for the loss of information. Third, 19 trainings took place after the COVID-19 pandemic, with 17 being online and 2 being face-to-face, compared to 40 all being face-to-face prior to the pandemic. This could potentially skew the results as it relates to modality and perceived training value. Fourth, because responses from mentees and mentors could not be cross-referenced post-training, it is impossible to determine if the intent to change resulted in actual behavior change by participants. Finally, there were 41 different trainers that conducted the 59 trainings in this analysis, and the effectiveness of facilitators was not included in this analysis.

Future research

This study provides several directions for future research. First, more investigation is needed to understand the impact of modality on mentor and mentee training. Rogers et al. conducted their study [26] on the effect of modality on training using a sample from 2015 to 2018, while the sample in this study came from 2015 to 2022, with 17 trainings taking place online during the COVID-19 pandemic. It is possible that the COVID-19 pandemic influenced participant perceptions of online education. Further research is required to determine if the lower perceived value of online education persists, as more and more educational interventions move online in the post-pandemic world [32]. Second, as many mentor scholars have suggested, more longitudinal research is needed to understand the mid- and long-term influence of Mentoring Up training on the behavior of mentees and mentors. Third, much more research is needed in mentoring scholarship analyzing responses from both mentor and mentee to see how training impacts the relationship as it progresses through different stages of development and in different training and professional contexts (e.g., graduate student-professor, junior professor-professor, etc.). Fourth, this study supported previous research that showed the individuals with prior mentor training perceived less value and smaller changes in self-assessed mentoring competence compared with those participants with no prior training. Future scholarship should seek to develop training that feels more summative or synergistic for participants who have experienced prior mentor training. Fifth, this student did not measure participant satisfaction with their current (most relevant) mentor. Further research is needed to understand how satisfaction with the most relevant mentoring relationship impacts intent to change behavior in Mentoring Up training. Finally, the study does not directly distinguish between learning gains that participants wish to implement now as mentees or in the near future as mentors. Particularly for those who are at transitional stages in their careers, the curriculum invites them to think more intentionally of their roles as both mentors and mentees. The curriculum and its facilitators would benefit from research that studies how and when participants are choosing to implement behavior changes and in which roles as they transition toward more independent research careers.

Summary

This study suggests that participants almost universally improved their mentoring competence by participating in the Mentoring Up curriculum. Of the factors that were thought may influence the impact of the curriculum, modality, previous mentor training, finding value in the training, and intent to change were all significantly associated with the magnitude of self-evaluated change in mentoring competence. Participants who engaged in face-to-face training were far more likely than online trainees to report higher levels of change in mentoring competence. In addition, online participants were far less likely to see value in the training. This was additionally significant as the value of training was associated with the magnitude of change to self-perceived mentoring competence. Finally, intent to change behavior was significantly and practically associated with a self-perceived change in mentoring competence. Participants with the largest self-reported skill gains were far more likely to report change behaviors that were emphasized in the training, most frequently citing maintaining effective communication and aligning expectations.