Measuring organizational mentoring climate: Importance and availability scales

Abstract

Introduction:

Although organizational climate may affect faculty’s mentoring behaviors, there has not been any way to measure that climate. The purpose of this study was to test the reliability and validity of two novel scales to measure organizational mentoring climate importance and availability at two public research universities.

Methods:

We developed 36 content-valid mentoring climate items in four dimensions: Structure, Programs/Activities, Policies/Guidelines, and Values. In total, 355 faculty completed an anonymous, structured, online survey asking about the importance (very important to very unimportant) and availability (no, don’t know, yes) of each of the items. We conducted reliability analyses and construct validity testing using exploratory common factor analysis, principal axis factoring, and oblique rotation.

Results:

The majority of the predominantly female, White non-Hispanic, senior, tenure-track faculty were not currently mentoring another faculty or being mentored. Analyses demonstrated a 15-item solution for both the Organizational Mentoring Climate Importance (OMCI) and the Availability (OMCA) Scales, with three factors each: Organizational Expectations, Mentor–Mentee Relationships, and Resources. Standardized Cronbach alphas ranged from 0.74 to 0.90 for the subscales, and 0.94 (OMCI) and 0.87 (OMCA) for the full scales. Faculty rated all items as somewhat to very important; however, perceived availability was very low ranging from mentor training programs (40%) to guidelines for evaluating mentoring success or managing conflict (2.5%).

Conclusions:

The scales will allow studying of how organizational climate may affect mentoring behavior and whether climate can be changed to improve faculty mentoring outcomes. We provide recommendations for furthering the science of organizational mentoring climate and culture.

Introduction

Evidence suggests that mentoring improves professional outcomes for both faculty mentors and mentees. In descriptive correlational studies, mentorship has been associated with improved mentee productivity [1–6], promotion [1,2,7], academic self-efficacy [8–11], likelihood of mentoring others [6], and career satisfaction [12–14]. In a systematic review, mentorship has also been linked to personal development, career guidance, and career success [3]. Furthermore, in the only study with a comparison group, mentored faculty’s mean grant dollars and proposal counts were higher than the comparison cohort [15]. Thus, the preponderance of evidence and anecdotal confirmation reveal the implicit impact of mentoring for faculty career development.

Individual faculty behavior, such as serving as a mentor, is likely influenced by individual beliefs, knowledge and actions; social network expectations and behaviors (perceived or real); and characteristics of the organization in which the individual works (Fig. 1) [16–19]. Organizational climate is defined as the shared perceptions of and the meaning attached to the policies, practice, and procedures employees experience and the behaviors they observe that are supported, expected, or rewarded [20]. In general, climates arise due to three distinct processes within organizations: exposure to tangible structural characteristics (e.g. committees); practices that lead to the attraction, selection, and retention of people with similar characteristics; and socialization that teaches appropriate behavior [21–23]. Organizations do not have single climates; multiple climates co-exist, each related to various organizational processes and strategic outcomes [20]. For example, the climate for safety may be different from the climate for mentoring.

Fig. 1. Individual and organizational influences on mentoring behavior and outcomes (adapted from [17]).

While there are measures of individual outcomes related to research mentoring [24], no reliable and valid scales currently exist for assessing the importance and availability of components of the organizational mentoring climate at research institutions of higher education [16,25]. There is also no information about which components of mentoring climate are relevant to faculty, whether the content or quality of organizational support for mentoring is related to mentoring behavior, and how to measure the impact of climate interventions. The lack of reliable and valid scales to assess the organizational mentoring climate constitutes a critical gap in studying interventions to advance the climate, which is crucial in order to achieve the long-term goal of creating and strengthening a skilled and diverse academic workforce. Our study’s objective was to perform psychometric testing of two novel scales to measure organizational mentoring climate importance and availability at research institutions of higher education.

Materials and Methods

Sample

In this cross-sectional, psychometric study, we invited a study population of 6152 faculty from two public, research-intensive universities in the southwest United States, one with 3195 faculty across main, health sciences, and branch campuses, and the other a university without a medical school with 2957 faculty. A total of 616 (10.0%) surveys were returned; 355 (5.8%) had complete data (defined as answering at least one item near the end of the survey; 8.4% from University #1 and 2.7% from University #2) and constituted the final sample for psychometric analyses. The sample size for the initial 36-item scales was adequate with 5–10 participants per item (per scale) using accepted techniques for sample size calculations for exploratory models of factor analysis when statistical significance is not being tested and the concept of power does not apply [26–29].

Procedures

Procedures for instrument development included item development based on the literature and investigators’ experience, the evaluation and rating of content validity, and final administration to faculty to evaluate the reliability and construct validity of the Organizational Mentoring Climate Importance (OMCI) Scale and the Organizational Mentoring Climate Availability (OMCA) Scale.

Item development

We developed 37 mentoring climate items classified in four dimensions: Structure (13 items), Programs/Activities (11 items), Policy/Guidelines (9 items), and Overall Value (4 items). Items in the first three dimensions were based on Keyser and colleagues’ [16] conceptual model of institutional efforts to support academic mentoring. We supplemented those items with additional items from a literature review and our own experiences, particularly with mentoring faculty from under-represented populations.

Content validity

Content validity of the items was determined using established methods [30]. Four independent, nationally-known content experts from the National Research Mentoring Network [31] with expertise in mentoring and mentoring climate (all from outside the institutions where items were developed and construct validity was studied) participated in this phase. The experts separately evaluated whether each item was relevant to the overall conceptual framework of organizational mentoring climate and was related to the dimension in which it was placed in the survey, using an ordinal 4-point rating scale (1 = not relevant; 2 = unable to assess relevance without revision or needs such a revision that it would no longer be relevant; 3 = relevant but needs minor alteration; 4 = very relevant and succinct). Based on the content validity process, 3 of the 37 items were deleted, 2 new items were written, and 8 were modified based on feedback. The remaining 36 items were judged as content valid and retained. The mean content validity scores for the individual items as they related to the overall conceptual framework ranged from 3.25 to 4.0 [32]. The number of items in the developed instrument was consistent with guidelines that an instrument’s initial item pool be composed of 1.5 to 2 times as many items as the final instrument [33].

Instrument testing

We created email distribution lists using faculty rosters from both universities with permission from the appropriate institutional offices and distributed email invitations to participate in the online Research Electronic Data Capture (REDCap) survey [34]. The invitation clearly stated that participation in the study was purely voluntary. Faculty were provided a web-linked option in the email to accept or decline participation. Those who did not decline or complete the survey were sent additional bi-weekly invitations over 4 months using the automated REDCap survey administration feature. Those who started but did not complete the survey were provided a reminder to finish.

The online survey included an IRB-approved informed consent document, with an implicit consent if the participant clicked on the link provided in the email. The link provided access to the anonymous, structured, self-administered REDCap survey. The survey took approximately 20 min to complete. The first scale (OMCI) asked about the importance of 36 items (organizational structures, programs, policies, and values related to mentoring) using 5-point Likert response options ranging from “very important (1)” to “very unimportant (5)” and a neutral midpoint. A sample importance item is: “Rate how important you think the following is to mentoring success, in general, at any institution – An annual award for excellence in mentorship to a faculty member.” The second scale (OMCA), located in a different section of the survey, used the same 36 items but asked respondents about the presence or absence of the item at their college/school/department/division (30 items) or their institution (6 items). Response options were “no (−1),” “don’t know (0),” and “yes (1).” We also included a brief set of demographic questions including gender, race, ethnicity, career stage, institution, faculty track and rank, and experience with mentoring.

Data Analysis

For construct validity testing of both the OMCI and the OMCA Scales, we used exploratory common factor analysis with principal axis factoring and oblique Promax rotation, assuming that each item was composed of both systematic and random measurement error and that factors would be correlated. Because a higher correlation makes it more difficult to interpret the factor structure loadings, delta (δ) was set at −1.0 in SAS software [35] to limit the degree of correlation (obliqueness) between factors to approximately 0.30 [33,36]. Factor extraction followed guidance that extraction be continued until all extracted factors account for at least 90% of the explained variance or until the last factor accounts for only a small portion (<5%) of the explained variance [37], as well as Eigen value and scree plot examination. We examined factor pattern matrices to determine simple structure for both scales as these matrices control for correlation among the factors and are most easily interpreted [37,38]. Criteria for retaining items included (a) rounded loading of ≥0.40 on a given factor; (b) loading is at least 0.10 to 0.15 higher than loading on any other factor; and (c) theoretical considerations.

We assessed internal consistency reliability using Cronbach’s alpha coefficients. We made final decisions about item retention and subscale composition by examining inter-item correlation matrices, average inter-item and item-to-total correlation coefficients, and alpha estimates if the items were dropped [39].

Results

Sample

Table 1 shows the characteristics of the 355 respondents from the two universities. The sample was predominantly female (62.5%), White (76.3%), non-Hispanic (77.5%), and on tenure-track (54.9%) from non-health science campuses (54.2%). In total, 54.6% of the respondents were at senior faculty ranks of Associate or Full Professor, evenly split across the individual ranks. The majority were not involved in a faculty mentoring relationship as mentors (59.7%) or mentees (73%) at the time of the study.

Table 1. Respondent characteristics (n = 355)

Item Analysis and Descriptive Statistics

Responses for each of the final 15 items (from the 36 initial items) ranged from 1 (very important) to 5 (very unimportant) for the OMCI; and −1 (no) to 0 (don’t know) to 1 (yes) for the OMCA Scales. The means of the individual items for the OMCI Scale (Table 2) ranged from 1.5 to 2.1 (SD = 0.8–1.1), indicating a general perception that all items were at least somewhat important.

Table 2. Descriptive statistics of 15-item Organizational Mentoring Climate Importance (OMCI) scale, and Organizational Mentoring Climate Availability (OMCA) scale items

¹ Rate how important you think each of the following are to mentoring success, in general, at any institution. Responses: 1 = Very important; 2 = Somewhat important; 3 = Neither important nor unimportant; 4 = Somewhat unimportant; 5 = Very unimportant.

² Available at my college/school/department/division (items 1–11) or institution (items 12–15). Responses: −1 = No; 0 = Unsure; 1 = Yes.

³ Remaining percentages to equal 100% from those who did not respond (only 0–3 respondents).

For the OMCA Scale, the means ranged from −0.65 to 0.16 (SD = 0.54–0.88), indicating that the perceived availability of the 15 items was generally low. At one extreme, 39.7% of respondents reported that their institution had a mentor training program, 36.6% had mentorship training materials, 29% discussed mentoring requirements at meetings, and 26.2% had a policy or guideline providing access to unconscious bias training for all faculty. On the other extreme, only 2.5% reported any type of policy or guideline to manage conflict in the mentor–mentee relationship; furthermore, as little as 2.3% reported the availability of either a policy or a committee delineating the criteria to evaluate mentoring success. In total, 14.9% to 60.8% of respondents reported that they did not know about availability of the 15 items.

Exploratory Factor Analysis

Matrix analysis demonstrated that factor analysis was appropriate to evaluate the data for both the OMCI and OMCA items. For the OMCI Scale, the Kaiser–Meyer–Olkin measure of sampling adequacy (MSA) was 0.95 and all MSAs for the individual items were higher than 0.90. For the OMCA Scale, the Kaiser–Meyer–Olkin MSA was 0.88 and all MSAs for the individual items were greater than 0.82.

We wanted to develop two parallel scales (OMCI and OMCA) with identical items for administration and interpretation ease. Our analysis began with 36 importance items. After examination of the correlation matrices, we dropped eight items highly correlated (r > 0.79) with similar items. Initial principal axis factor analysis with oblique rotation of the remaining 28 items suggested a three-factor solution and resulted in the elimination of four items with weak loadings (<0.40) and two items with cross-loadings separated ≤0.10 on two different factors. All six deleted items had other retained items with similar content.

We conducted factor analysis, using a three-factor solution, of the 22 OMCA items corresponding to the 22 remaining OMCI scale items to ascertain whether there was a similar factor structure for the two scales. Fifteen OMCA items loaded identically to the same OMCI scale items, resulting in the final two 15-item scales presented in this paper. Of the final seven items dropped from both scales based on the OMCA analysis, one item did not load on any factor, two had cross-loadings, and four loaded on different factors than the OMCI analysis. All seven deleted items had other retained items with similar content.

The final exploratory analyses showed a three-factor solution for each of the two scales that explained 94% and 88% of the variance for the OMCI and OMCA Scales, respectively. Table 3 shows the final factor pattern matrices for the 15-item scales and their three subscales: Organizational Expectations, Mentor–Mentee Relationships, and Resources. Loadings are similar to partial standardized regression coefficients in multiple regression.

Table 3. Oblique rotated (PROMAX) factor pattern matrices, means, standard deviations and standardized Cronbach alpha reliabilities for 15-item Organizational Mentoring Climate Importance (OMCI) and Availability (OMCA) scales and subscales

Internal Consistency Reliability

Table 3 shows the means, standard deviations, and standardized Cronbach alpha internal consistency reliabilities for the two scales and their three subscales. Cronbach alpha values ranged from 0.74 to 0.90 for the subscales, a very acceptable range for newly developed instruments [26,33]. Cronbach alpha values for the full 15-item OMCI and OMCA Scales were 0.94 and 0.87, respectively.

Discussion

For decades, faculty at research institutions of higher education have contended that mentoring is critical to faculty success [17,18,25,31]. Although organizational climate may affect faculty’s mentoring behaviors, until now, there has not been a research-based instrument to measure organizational mentoring climate. The results of this study provide evidence of the preliminary reliability and validity for the newly developed OMCI and OMCA Scales designed to measure the importance and availability of critical organizational structures, programs, and policies related to mentoring. Measures like these are essential for studying whether organizational climate is associated with mentoring behavior and if so, what interventions might be effective in changing the organizational mentoring climate.

We chose the items in this study based on the literature and our experience-based hypotheses about which organizational characteristics are important for mentoring climate; we empirically tested these items with faculty to determine perceived importance and availability. Based on this data, we posit that organizations that desire to have a positive organizational mentoring climate should invest in elements in three areas: organizational expectations related to mentoring involvement, guidance in formalizing the mentor–mentee relationship, and concrete resources to support mentoring. These investments would entail creating explicit expectations for senior faculty’s engagement in mentoring relationships with junior faculty; establishing opportunities for the regular discussion of specific qualifications to be an effective mentor; implementing structures for the frequent examination of the quality of the relationships; and including mentoring success as a criterion for tenure and promotion. There is a need to formalize the mentor–mentee relationship by assessing the mentors’ qualifications and mentor–mentees’ assignments; establishing the criteria to evaluate the success of the relationship; and, providing guidance and structures for managing conflict between mentors and mentees. Finally, mentor training is a major initiative identified for improving mentoring in academia [17,40] and the results of this study support that emphasis; mentor training programs and materials are identified as concrete resource components of organizational mentoring climate. Resources to support mentoring of under-represented groups are also crucial [41,42].

There is a glaring discrepancy between the perceived importance of all organizational mentoring climate characteristics and their actual availability. What is considered important is often not available. These data suggest that research institutions of higher education may be slightly better in providing concrete resources than setting explicit expectations or formalizing mentor–mentee relationships. Even if one assumes that some of the respondents did not know that specific resources are available at their institution, the lack of awareness in itself is a concern. Further study is indicated to determine the accuracy of faculty’s perceptions of institutional mentoring supports relative to the actual availability of those resources. However, the results from this study are consistent with other reports demonstrating that less than half of the studied universities or departments require mentoring for faculty promotion [5,43]; have formal mentor training [43,44]; or have criteria to qualify as a mentor, stated mentor responsibilities, mentor/mentee written agreements, or guidelines for conflict management [44]. It is not surprising then that nearly two-thirds of faculty report that they are not currently involved in mentoring relationships with other faculty [43]. Only one-third to one-half of faculty report having a mentor; in some fields, the prevalence is as low as 20% [3,11,12,45], and overall, less than half of the faculty surveyed have had sustained, influential mentoring relationships [6].

Based on the results of this study, we offer several recommendations for advancing the science of faculty mentoring. First, we found beginning support for the reliability, content, and construct validity of the newly developed OMCI and OMCA Scales; however, construct validity must be further developed with additional diverse faculty samples from multiple institutions, using confirmatory factor analysis to test for evidence of replicated factor structure. Test–retest reliability, convergent, and discriminant validity must also be explored.

Second, we need to assess whether activities that address specific gaps in organizational mentoring climate impact faculty mentoring success and whether our measures of mentoring climate are sensitive or responsive to organizational interventions designed to improve that climate. Little agreement exists about organizational characteristics that will encourage, improve, or reward faculty mentoring behavior and relationships [16,25,44]. In studies of climates other than mentoring, not all demonstrate a significant moderator effect for climate strength in predicting outcomes, so merely counting the number of available mentoring climate items at an institution may be inadequate [20]. We may need to explore more complex causal models related to desired outcomes (see Fig. 1).

Third, consistent with existing measures of organizational climate, the scales measure individual faculty’s perceptions of the mentoring climate; however, there may be aggregate effects, such as school- or department-, or division-level effects, that are different from the individual. Microclimate or the climate of the organization most proximal to an individual may be more important than the climate of the overall parent organization. In this study, we asked participants to focus on the level most relevant to them when answering questions related to availability. Items in the availability scale that were linked to the college/school/department/division levels rather than the institutional level (the resource items) were also items with the highest ratings on the importance scale. We recommend that future studies pay attention to the level at which the mentoring climate is measured. Examination of inter-rater agreement, including both within and between-group inter-rater reliability in future studies would provide support for aggregation of individual perceptions into group or organizational levels of analysis [20]. Studies using multilevel modeling are ultimately necessary to discriminate between individual, dyadic, group, and organizational influences on mentoring behavior.

Finally, while there is some debate, we need to carefully consider whether or not organizational mentoring climate is the same as mentoring culture. Culture is defined as: “the shared values and basic assumptions that explain why organizations do what they do and focus on what they focus on; it exists at a fundamental, perhaps preconscious level of awareness that is grounded in history and tradition” (p. 468) [46]. In this study, we made a distinction between organizational climate and organizational culture related to mentoring. Organizational climate includes factors that are more visible and easier to assess and change. In contrast, organizational culture includes the unspoken rules, values, and beliefs that may influence faculty behavior and engagement [20,25,46]. Organizational mentoring culture is likely initiated and reinforced by the organizational climate – the structures, programs, and policies related to mentoring. In this study, the four values items developed in the content validity phase did not load in the final factor analyses; this suggests that either they were integral to the final remaining items and did not contribute unique variance or that they represented some other incompletely measured construct such as organizational culture. Organizational leaders play a crucial role in establishing and reinforcing a values-based organizational culture [21], and it is likely that both the climate and culture set the tone for the importance, value, rewards, and consequences of mentoring [18,45,47,48].

This study has several limitations. First, although we sent multiple email invitations, only a small number of potential participants responded to the survey. Despite that, the number of participants was sufficient to do the planned psychometric testing. Psychometric studies typically require 1.5–2 times as many initial items as the items in the final instruments [33]. Our survey was lengthy due to the need to test 36 items for two different scales. The possibility that length may have affected response is based on the observation that while 616 (10%) of surveys were returned, only 355 (5.8%) respondents completed an item toward the end of the survey and were usable. In addition, response was highest at the lead investigator’s institution (8.4% vs. 2.7%). We anticipate that shorter surveys (15-item scales), ideally sent from an investigator affiliated with the study university, would improve response rates in the future.

In addition, respondents were predominantly female (62.5%), White (76.3%), non-Hispanic (77.5%), and slightly more than half were on the tenure-track (54.9%). Female, Whites, and Hispanics on the tenure-track were over-represented in the sample when compared to the university populations. (For comparison, University 1: 50.8% female, 62.8% White, 85.6% non-Hispanic, 31.3% tenure-track [49]; University 2: 45.4% female, 73.2% White, 92.4% non-Hispanic, 42.1% tenure-track [50,51]). Just as construct validity of the scales must be tested in additional samples, future subgroup analyses are warranted to determine if perceptions of climate are the same across different demographic and professional groups. One could speculate that female, Whites, or Hispanics on the tenure-track may be more likely to respond to a survey about, and have heightened interest in, mentoring climate when compared to individuals in other groups. Finally, some respondents may have had limited knowledge about certain organizational characteristics related to mentoring, possibly reflecting the relatively high number of “I don’t know” responses. This perceived limited availability may reveal either a lack of high-level initiatives or designated leadership support for the promotion of mentoring activities at the institution.

Conclusions

This research addresses a new area of science, the study of organizational mentoring climate in universities. For the first time, there are self-report scales that may be used to describe and quantify both the availability of and perceptions about the importance of specific mentoring structures, programs, and policies; this will allow further studies about how organizational mentoring climate may affect mentor–mentee behavior and whether it is possible to change organizational mentoring climate to improve faculty mentoring outcomes.

Changing the organizational mentoring climate is not easy; however, it could have important ramifications for faculty development and sustainability, particularly the career advancement of individuals from under-represented groups [25]. Change requires a shared vision, energetic change agents, significant long-term support from institutional leaders, and widespread buy-in from all faculty. A healthy organizational mentoring climate is essential for achieving the long-term goal of a skilled and diverse academic workforce.