Monitoring and evaluating capacity building activities in low and middle income countries: challenges and opportunities

Abstract

Background

Lower and middle income countries (LMICs) are home to >80% of the global population, but mental health researchers and LMIC investigator led publications are concentrated in 10% of LMICs. Increasing research and research outputs, such as in the form of peer reviewed publications, require increased capacity building (CB) opportunities in LMICs. The National Institute of Mental Health (NIMH) initiative, Collaborative Hubs for International Research on Mental Health reaches across five regional ‘hubs’ established in LMICs, to provide training and support for emerging researchers through hub-specific CB activities. This paper describes the range of CB activities, the process of monitoring, and the early outcomes of CB activities conducted by the five research hubs.

Methods

The indicators used to describe the nature, the monitoring, and the early outcomes of CB activities were developed collectively by the members of an inter-hub CB workgroup representing all five hubs. These indicators included but were not limited to courses, publications, and grants.

Results

Results for all indicators demonstrate a wide range of feasible CB activities. The five hubs were successful in providing at least one and the majority several courses; 13 CB recipient-led articles were accepted for publication; and nine grant applications were successful.

Conclusions

The hubs were successful in providing CB recipients with a wide range of CB activities. The challenge remains to ensure ongoing CB of mental health researchers in LMICs, and in particular, to sustain the CB efforts of the five hubs after the termination of NIMH funding.

Background

Capacity building (CB) in mental health is a strategy to improve mental health systems, services, and research in low- and middle-income countries (LMICs). CB will contribute towards reducing the treatment gap in these countries (Patel et al. 2011; Tol et al. 2011; Thornicroft et al. 2012). The need for CB in LMICs is underscored by the burden of mental, neurological, and substance use disorders (Steel et al. 2014; Global Burden of Disease Collaborators, 2015), and the proportionately low representation of research publications from lead authors in LMICs in leading mental health journals (Patel & Kim, 2007; Sharan et al. 2007).

The 2013 Global Burden of Disease study (Global Burden of Disease Collaborators, 2015) highlights mental disorders as leading causes of years lost to disability. This burden falls heavily on LMICs – home to 85% of the world's population (Sharan et al. 2009). Yet the research output regarding mental health is comparatively low for LIMCs. Only 6% of publications on ISI Web of Science (from 1992 to 2001) (Saxena et al. 2006) and 3.8% of publications in six prominent psychiatric journals from 2002 to 2004 (unchanged from 1996 to 1998 results) (Patel & Kim, 2007) were from lead authors in LMICs. Reasons for limited published mental health research and limited mental health services in LMICs include: (i) high levels of stigma towards mental disorders, (ii) lack of sufficient and/or appropriate professional development opportunities in mental health research, (iii) limited funding for and culture of research, (iv) few skilled researchers, and (v) few collaborative endeavours (Sharan et al. 2007; Sweetland et al. 2014).

CB initiatives help to address these barriers through strategies such as (i) south-north and south–south collaborations (Sweetland et al. 2014), (ii) harnessing collaborative initiatives to produce high quality research data, and (iii) using twenty-first century technology that makes information and distance learning more accessible than ever before. While it is important to develop CB initiatives, it is also important to incorporate a monitoring and evaluation (M & E) component to ensure that these initiatives achieve their CB aims.

The National Institute of Mental Health (NIMH) has funded five collaborative hubs for mental health research – two in Latin America – Regional Network for Mental Health Research in Latin America [RedeAmericas (Susser et al. 2012] and Latin American Treatment and Innovation Network in Mental Health (Latin-MH), two in sub-Saharan Africa – Partnership for Mental Health Development in sub-Saharan Africa (PaM-D) and Africa Focus on Intervention Research for Mental Health (AFFIRM) (Lund et al. 2015) and the South Asian Hub for Advocacy, Research and Education on Mental Health (SHARE) hub in south Asia. The aim of these hubs is to develop research on task sharing interventions for mental health, develop researcher networks, and build capacity in mental health research in their respective regions. The CB strategies developed by the five hubs has been described in a sister paper (Pilowsky et al. 2016) and in the specific hub papers in this special issue. As suggested by Sharan et al. (2007), mapping and monitoring CB activities should be an important component of CB strategies. This paper aims to describe and monitor the outcomes of CB activities for research conducted by five research hubs to showcase the range of feasible CB activities and provide initial indications for successful strategies.

Methods

A basic M & E framework was developed from which the common indicators were extracted for comparison of the CB activities and outcomes across five hubs.

Development of the M & E framework

Firstly, every hub compiled a list of their CB opportunities. This list described the content, target audience, application process, cost, attendance, and outcomes for each activity. Secondly, each hub set out their main indicators (including the number of student-initiated publications, number of courses and workshops with attendance and completion rates, and grant applications submitted) to be used for M & E the outcomes from these CB activities. The second author evaluated these lists to identify the common indicators, which were subsequently set into standardized forms for data capturing within hubs on an ongoing basis. Table 1 illustrates the two categories for the common indicators used for the cross-hub analysis presented in this paper, namely (i) CB activities and (ii) monitoring of outputs.

Table 1. Comprehensive list of the indicators used for Monitoring and Evaluating the capacity building (CB) strategies of the five collaborative hubs

^a Intra-hub activities are usually multi-country in focus but within one hub, while inter-hub activities involving more than one hub.

Description of common indicators

CB activities were mostly centred around workshops and courses. Some of these CB activities were available only to participants within a hub (intra-hub) while others were available for participants across the hubs (inter-hub).

Apart from immediate course evaluation surveys examining the subjective experience and perceived benefit of the CB activity, hubs also monitored subsequent outputs of participants. For example, in the PAM-D hub former CB participants were regularly followed up via email to monitor their publication outputs, grant applications, and career progression, while for the AFFIRM hub a brief survey was sent to MPhil graduates towards the end of 2015.

Although some of the hubs obtained Institutional Review Board (IRB) approval for the overall hub project, others only obtained IRB approval for the randomized controlled trials (RCT) run within the hub. No IRB approval was obtained specifically for the CB activities from any of the hubs as this did not involve research with human subjects.

Results

The results from the M & E exercise for the five hubs combined are divided into two sections: (i) CB activities (attendance and completion rates for both intra-hub and inter-hub activities) and (ii) M & E indicators (publications, grant applications, and career tracking). As the paper describes the trends across hubs, the detail is limited to an overview with additional detail available in the other hub-specific papers in this special issue.

CB activities

Intra-hub courses and workshops

Table 2 sets out the individual courses for each hub and how many fellows or awardees were selected and completed the training. As the PaM-D and Latin-MH hubs started later, they have completed fewer activities than the other three hubs.

Table 2. Attendance and completion for capacity building (CB) activities of the five Collaborative Hubs for International Research on Mental Health

Overview of CB activities: All five hubs ran courses and workshops from the first year. Evident in Table 2, these varied extensively in time range and content, from full postgraduate degree courses to short face-to-face or online courses and mentorships. Where possible, presentations were made available online after the CB activity.

Overall completion rates: The completion rate for the courses across all hubs is high (ranging between 75% and 100%). By nature, short courses are more feasible to complete and demonstrated highest completion rates. Short courses were run by internationally recognized experts in the field. Although still high, courses with longer duration tended to have lower completion ratios than short courses. The high completion rates are not unexpected as the stringent nature of the selection process together with good funding (Table 2) ensured that participants were motivated and able to attend and complete the courses, and participants had access to support systems and mentoring through the allocation of course supervisors or mentors as evident by examples from different hubs.

AFFIRM runs the MPhil in Public Mental Health. This is a 2 year full time course providing supervision (primarily by the Stellenbosch University or University of Cape Town with in-country co-supervisors) to students from five sub-Saharan African countries. Most of the students are working full-time while completing their dissertation. This, coupled with various challenges in maintaining good contact between students and supervisors, has meant that a number of students have either not completed or taken longer than the 2 year funded period, which requires that they find their own funding for the final year. In addition, AFFIRM hosts Ph.D. students linked to the RCT and ran three short courses (see Table 2 and AFFIRM specific paper in this volume for further information).

Latin-MH provides national and international mentorships to its fellows. All fellows are supported in writing their own papers and to participate in the hub's publications. The young researchers, who are invited to work in the main project, have a ‘hands-on’ approach to activities related to the preparation and fieldwork of the hub's RCT. Some are also supported in writing applications for formal postgraduate training and grant applications. All fellows have mandatory activities provided by the hub that need to be developed during their fellowships and are monitored in their activities and their development. Postdoctoral fellows develop their own research within the main goals of the hub.

PaM-D provides mentoring and supervision through a number of CB activities. (1) The Mental Health Leadership and Advocacy Programme (mhLAP) targets future mental health leaders and advocates from five English speaking countries in West Africa (Nigeria, Ghana, Sierra Leone, Liberia and Gambia) as well as the other two hub countries, Kenya and South Africa. Overall, the course was positively received with participants indicating that they welcomed the content focus on mental health legislation, policy, advocacy, prevention and promotion of mental health, indices of disease burden, as well as interacting with mental health practitioners from across Africa. (2) Two biostatistics short courses (beginners and intermediate) spanning 5 days were held in Cape Town, South Africa in mid-2014 with four senior biostatisticians/statisticians facilitating the course and 22 early career researchers from the five hub countries in attendance. (3) In March 2015, PaM-D hosted a three and half day applied writing workshop in Johannesburg, South Africa. Participants found the workshop very beneficial overall, with practical writing exercises particularly useful. The workshop facilitator continues to mentor workshop participants even months afterward. (4) Finally, an early career fellow within the Hub undertook a 2-week research mentored attachment in February 2016 at the Institute of Psychiatry, Kings College London, to acquire skills in systematic review writing. A second fellow will be undertaking a similarly focused attachment at the University of Stellenbosch, Cape Town, South Africa in 2016.

RedeAmericas provides support for young mental health professionals from Latin America who have shown potential to become successful independent investigators in projects related to the mental health needs of the region. Each awardee has a personalized, flexible training program. Core elements in the training program for most awardees are summer courses at Columbia University and University of Chile, and a mentoring program that includes the assignment of two mentors – one from Latin America and one from Columbia University. In addition, the majority of awardees participate in international and regional conferences, many of them play key roles in organizing these conferences, and through such activities they develop a self-sustaining regional network (which also includes non-awardees with similar interests). For example, awardees were central to the organization of three regional conferences: a Latin American Congress of Public Health in Cordoba, Argentina; a Latin American Congress of Epidemiology in Medellin, Colombia; and a Latin American Congress on Autism Spectrum Disorders in Santiago, Chile. Awardees are encouraged to pursue higher degrees such as MPH or Ph.D., and then to take academic positions as Assistant and later Associate Professors, which enables them to pursue research careers in public mental health. The mentors facilitate the progress of the awardees through all these CB activities including the design of the awardees’ first research proposal.

SHARE senior researchers provide guidance and mentoring to junior researchers to develop a career in mental health research, three SHARE researchers have enrolled in Ph.D. programs and embedded their doctoral project in ongoing SHARE projects. One SHARE fellow has won a grant to scale up his fellowship study in Sri Lanka. The SHARE core team also provides support to researchers and fellows in disseminating their work to a larger audience. All SHARE fellows have presented the results of their studies at annual SHARE meetings. Some of them were supported to present their study at the Global Mental Health Summit (hosted by SHARE) held in India in November 2015.

Inter-hub courses and workshops

In addition to the hub-specific CB activities described above, a number of the CB activities were also available to other hubs. The supercourse currently being developed jointly by the SHARE and RedeAmericas hubs is openly available for all hubs to use on an ongoing basis (http://www.pitt.edu/~super1/GMHS/index.htm). Other CB activities had limited participation from other hubs due to a limited number of places available on the course (e.g. limited places on the MPhil in Public Mental Health and the Implementation Science distance learning course) and geographical location of the courses (e.g. biostatistics course in Cape Town by PaM-D and RCT courses by AFFIRM in Cape Town and Addis Ababa). The materials from many of the courses are available on the hub websites (e.g. RCT course materials on the AFFIRM website). The SHARE hub ran a 6 week online course on implementation science and invited two participants from each of the other hubs. Table 3 presents a matrix of the courses run by the different hubs available and attended by other hubs.

Table 3. Summary of interhub capacity building (CB) activities across the five hubs

^a LATIN-MH is still at the early stages of the 5 year project and hence have little to report on as yet.

In addition to the CB activities run by the five collaborative hubs, the NIMH ran a number of webinars on topics identified by the hubs together in the regular CB workgroup teleconference meetings. The topics included complex interventions, presentations from researchers in the different hubs, and the range of funding opportunities available from the NIMH. These webinars were advertised widely by each hub and participation was open to all. Some challenges to participation were time zone differences and access to sufficient bandwidth to stream the webinar online.

One of the key benefits of inter-hub CB activities is the development and maintenance of a network of early and mid-career researchers in mental health across regions. Networks facilitate the development of collaborative projects and grant applications. Individual hub countries also provide opportunities for more localized networks to be developed, for example groups of students attending a course together.

At a different level and not specifically monitored, further CB opportunities are provided through each hub's annual meetings where all the hub countries come together and discuss the research being undertaken in the hubs. These annual meetings include CB activities or reports from fellows or students benefiting from them, or specific CB activities for the meeting participants. Participation in hub meeting discussions and presentations of the different intra-hub research activities contributes to CB of hub partners. The AFFIRM hub partners specifically commented on the benefit they received from. The AFFIRM hub partners specifically commented on the benefit they received from being part of the annual meeting discussions. The SHARE hub organizes a CB workshop with its annual meeting, including the SHARE researchers, fellows and public health practitioners working in the host country. So far, SHARE hub has organized such CB workshops on maternal mental health for local researchers in India, Bangladesh, and Sri Lanka.

Monitoring of outcomes

While the number of courses run, attendance numbers and completion rates are important outputs of planned CB activities, it is important to consider other more medium to long-term outcomes from the overall CB programs. These include reviewing the number of publications generated by participants in the CB activities, number of new grant applications submitted, and career pathways forged. Longer-term outcomes would include the reduction in the mental health treatment gap and sustainability of network within countries, regions and at a global level. This paper reviews the more short- and medium term outcomes of the CB activities of the five hubs. The publication describing the individual hub CB activities provides further information on individual hubs’ monitoring approaches (Pilowsky et al. 2016), as do the individual hub-specific papers in this special issue. The monitoring indicators implemented by each hub are presented in Table 4. All hubs prioritized publications, short course attendance, pre/post-course evaluations, and tracking of students as important indicators of outcomes. The other indicators (degree completion, participation in hub activities and progress reports for small grants were focused on hub-specific CB activities and were thus limited to the relevant hubs. Tracking of students has started but this is an ongoing and more long-term monitoring indicator that has not yielded much information as of early 2016.

Table 4. Monitoring indicators selected by each hub

Grants

Across all the hubs two local and six international grant applications were submitted, with SHARE and RedeAmericas focussing specifically on grant application as a CB activity. The SHARE hub has had three grant applications funded, two of which were developed with the assistance of CB fellows and one was prepared with a fellow as the lead. Further details can be obtained at the SHARE website (http://www.sharementalhealth.org).

RedeAmericas career awardees have had some success in receiving grant funding. Overall, six of the 13 awardees have received local grants as the principal investigator or co-investigator on a study since their training in the RedeAmericas program. Another awardee has recently submitted an application for a local grant to develop an RCT study and is waiting for approval and funding. For the other hubs grant proposals are anticipated as a medium to long-term outcome of increasing skills in mental health research.

Publications

All hubs used publication counts as a monitoring indicator. For this indicator we included only those publications where CB participants were lead authors. Across all hubs, a total of 13 international publications from individual hub students or trainees as lead authors have been accepted for publication. A number of further publications have been submitted for review or are in preparation (Table 4). The individual hub counts are as follows: (i) AFFIRM has one publication by a graduate from the MPhil programme (Udedi et al. 2013) and around four to five others in preparation; (ii) LATIN-MH – has three publications in preparation at the moment; (iii) PaM-D – one peer reviewed publication (Nortje et al. 2016), three under journal review and a number more in preparation; (iv) RedeAmericas has nine awardee-led peer reviewed publications with a number led by awardees in preparation; (v) SHARE has two publications led by fellows (Siriwardhana et al. 2013; Adhikari et al. 2015) and a number in preparation.

In RedeAmericas, there has been a strong focus on publication and each of the 13 awardees contributed valuable insight and information as authors on peer-reviewed publications. Throughout their time in RedeAmericas, they have collectively contributed to 61 articles published in local journals and 33 articles in international journals. Nine of these awardees have been listed as first author on a peer-reviewed publication. Most of their work has been published in local journals such as the Revista Argentina de Psiquiatria, the Jornal Brasileiro de Psiquitria, the Cadernos Saude Coletiva, and the Revista de Salud Publica. Three awardees have published peer-reviewed articles as first author in international journals including the Journal of Clinical Psychopharmacology, the Journal of Affective Disorders, Frontiers in Psychiatry, and the Archives of Psychiatry and Psychotherapy.

Tracking of students

This indicator considered career pathways and involvement in further degree training, grant writing, research, and publications by participants in the major CB activities such as mentorships and degree courses. The hubs reviewed dissemination of research information from students (e.g. in publications, blogs, or other formats) and tracking of career pathways on completion of their courses. For the AFFIRM hub, one of the earlier graduates from the MPhil has subsequently moved into a senior position in the Ministry of Health in his country. A number of AFFIRM MPhil students and graduates contributed to a blog developed as a lead-up to a panel session on the contribution of students to Global Mental Health presented at the International Mental Health Congress held in April 2015 in Lille, France (see https://storify.com/MHIN/studygmh). This blog was hosted by the Mental Health Innovation Network (MHIN).

The PAM-D hub requests regular progress and feedback from small grant awardees. In addition, the writing workshop candidates are closely followed up via email to determine the publication outputs upon completion of the course.

Discussion

The combination of the treatment gap, limited skills, lack of research, and few publications emanating from LMIC researchers in the field of mental health in LMICs requires a clear strategy to rectify the situation. Thornicroft et al. (2012) advocate addressing this through an increase in research funding, building capacity in mental health research through networks and collaborations, and fostering a culture of and incentives for mental health.

The importance of building partnerships through collaboration, networks, and mentorships between internationally recognized mental health research centres and developing researchers in LMICs is an important component of changing the visibility of mental health in LMICs (Fricchione et al. 2012; Sweetland et al. 2014). The allocation of a Latin American and US-based mentor for the RedeAmericas awardees is a good example of this, as was the use of internationally recognized mental health researchers in the numerous short courses run by the different hubs.

The CB strategies and related activities have as their aim the reduction of the treatment gap and an increase in mental health research in LMICs. All the hubs included aspects of research skills development and mentorship from both local and internationally recognized mentors, and included efforts to build sustainable networks. These networks were developed through regular meetings, webinars, and conference attendance for fellows, mentees or awardees.

The interest shown in all the hubs to participate in these CB activities (e.g. high number of applicants and good completion rates) attest to the growing interest in mental health research and in building knowledge and skills. These CB activities are well appreciated. However, monitoring of the impact of these CB activities is a medium to long-term endeavour and requires a ‘robust set of metrics’ (Cottler et al. 2015). As participants attend numerous CB initiatives it is challenging to identify which type of CB activity yields the most beneficial research output. Furthermore, over time we can consider the impact of different levels of research infrastructure or publication culture on outputs when comparing middle and low income countries and also determine whether the hubs made a difference to research productivity in the LMICs identified for inclusion. For example, repeats of the reviews cited at the start of this paper on the number of publication emanating from lead authors from LMICs (Saxena et al. 2006; Patel & Kim, 2007) could indicate changes over time that could be attributed to all CB strategies including of the hubs described in this paper.

Another medium to long-term challenge is expanding the research-training networks of the Hubs beyond individual researchers, groups, and academic institutions to include non-governmental organizations and policymakers. Effective monitoring of the success (quality) and sustainability of the mentorship of young trainees remains another challenge. In addition, other metrics such as memberships of professional societies, further degrees awarded, faculty appointments, and cross-cultural competencies gained should be specifically measured and monitored (Cottler et al. 2015).

The cost of the CB activities is high as it often requires much travelling, course registration fees, and costs of carrying out actual data collection. This does bring into question the sustainability of ongoing CB activities. But there are initiatives to fund ongoing CB and ensuring sustainability of these budding networks. One such initiative is the 5-year Welcome Trust funded CB project entitled Africa Mental Health Research Initiative (AMARI), which addresses CB in four countries (Ethiopia, Malawi, Zimbabwe and South Africa) at masters, doctoral, and post-doctoral level. The MHIN (see www.mhinnovation.net) provides a platform for blogs and dissemination of mental health research findings and comments from mental health researchers, practitioners, and people with mental disorders at a global level. Currently the MHIN has a global reach and is developing an African office (MHIN Africa) targeting this region.

Sustainability of CB initiatives will take time to develop with immediate costs being high until a critical mass of skills and research experience has been reached within individual countries and regions to sustain ongoing locally led development of CB, research, and grant applications.

In conclusion, the CB strategies and activities of the five Collaborative Hubs for International Research on Mental Health are proving successful in building research capacity within countries where it is most needed. The ongoing impact and sustainability of these efforts will emerge in the next decade. If the success continues, the impact will ensure culturally relevant and appropriate research that will address local needs (Saraceno & Saxena, 2004; Saraceno et al. 2007; Thornicroft et al. 2012) and building an evidence base for global mental health by contributing different geographical and cultural perspectives. The challenge remains to ensure ongoing CB of mental health researchers in LMICs to sustain the efforts of the five hubs.