Hostname: page-component-cd9895bd7-gbm5v Total loading time: 0 Render date: 2024-12-26T05:08:56.213Z Has data issue: false hasContentIssue false
  • English
  • Français

An Electronic Competency-Based Evaluation Tool for Assessing Humanitarian Competencies in a Simulated Exercise

Published online by Cambridge University Press:  21 February 2017

Andrea B. Evans ,
Jennifer M. Hulme ,
Peter Nugus ,
Hilarie H. Cranmer ,
Melanie Coutu  and
Kirsten Johnson
Andrea B. Evans
Affiliation:
Emergency Department, The Hospital for Sick Children, Toronto, Canada Center for Global Child Health, The Hospital for Sick Children, Toronto, Canada
Jennifer M. Hulme
Affiliation:
Emergency Department, University Health Network, University of Toronto, Toronto, Canada Department of Family and Community Medicine, University of Toronto, Toronto, Canada
Peter Nugus
Affiliation:
Centre for Medical Education, McGill University, Montreal, Canada Department of Family Medicine, McGill University, Montreal, Canada
Hilarie H. Cranmer
Affiliation:
Global Health, Massachusetts General Hospital, Boston, MassachusettsUSA
Melanie Coutu
Affiliation:
McGill Humanitarian Studies Initiative, McGill University, Montreal, Canada Humanitarian U Inc., Montreal, Canada
Kirsten Johnson*
Affiliation:
Department of Family Medicine, McGill University, Montreal, Canada McGill Humanitarian Studies Initiative, McGill University, Montreal, Canada Humanitarian U Inc., Montreal, Canada Emergency Department, McGill University Health Centre, Montreal, Canada
*
Correspondence: Kirsten Johnson, MD, MPH McGill University Department of Family Medicine 5858 Cote des Neiges, 3rd Floor, Suite 300 Montreal, Quebec, Canada H3S 1Z1 E-mail: Kirsten.johnson@mcgill.ca
Get access Rights & Permissions [Opens in a new window]

Abstract

Methods

The evaluation tool was first derived from the formerly Consortium of British Humanitarian Agencies’ (CBHA; United Kingdom), now “Start Network’s,” Core Humanitarian Competency Framework and formatted in an electronic data capture tool that allowed for offline evaluation. During a 3-day humanitarian simulation event, participants in teams of eight to 10 were evaluated individually at multiple injects by trained evaluators. Participants were assessed on five competencies and a global rating scale. Participants evaluated both themselves and their team members using the same tool at the end of the simulation exercise (SimEx).

Results

All participants (63) were evaluated. A total of 1,008 individual evaluations were completed. There were 90 (9.0%) missing evaluations. All 63 participants also evaluated themselves and each of their teammates using the same tool. Self-evaluation scores were significantly lower than peer-evaluations, which were significantly lower than evaluators’ assessments. Participants with a medical degree, and those with humanitarian work experience of one month or more, scored significantly higher on all competencies assessed by evaluators compared to other participants. Participants with prior humanitarian experience scored higher on competencies regarding operating safely and working effectively as a team member.

Conclusion

This study presents a novel electronic evaluation tool to assess individual performance in five of six globally recognized humanitarian competency domains in a 3-day humanitarian SimEx. The evaluation tool provides a standardized approach to the assessment of humanitarian competencies that cannot be evaluated through knowledge-based testing in a classroom setting. When combined with testing knowledge-based competencies, this presents an approach to a comprehensive competency-based assessment that provides an objective measurement of competency with respect to the competencies listed in the Framework. There is an opportunity to advance the use of this tool in future humanitarian training exercises and potentially in real time, in the field. This could impact the efficiency and effectiveness of humanitarian operations.

EvansAB, HulmeJM, NugusP, CranmerHH, CoutuM, JohnsonK. An Electronic Competency-Based Evaluation Tool for Assessing Humanitarian Competencies in a Simulated Exercise. Prehosp Disaster Med. 2017;32(3):253–260.

Keywords

competencyevaluationhumanitarian responsehumanitarian traininghumanitarian workerssimulation exercise
Type
Original Research
Information
Prehospital and Disaster Medicine , Volume 32 , Issue 3 , June 2017 , pp. 253 - 260
Copyright
© World Association for Disaster and Emergency Medicine 2017 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Conflicts of interest: Kirsten Johnson is the CEO of Humanitarian U. The other authors declare no conflicts of interest.

References

1. United Nations Office of Humanitarian Affairs. Global Humanitarian Overview 2016: A consolidated appeal to support people affected by disaster and conflict. Published 2016. https://www.humanitarianresponse.info/fr/programme-cycle/space/document/global- humanitarian-overview-2016-consolidated-appeal-support-people. Accessed January 28, 2016.Google Scholar
2. United Nations Population Fund. State of the World’s Population Report 2015. Published 2015. http://www.unfpa.org/sites/default/files/sowp/downloads/ State_of_World_Population_2015_EN. pdf. Accessed January 28, 2016.Google Scholar
3. Department of International Development -United Kingdom. Humanitarian Emergency Response Review. Published 2011. https://www.gov.uk/government/uploads/ system/uploads/attachment_data/file/67579/HERR.pdf. Accessed January 4, 2016.Google Scholar
4. United Nations Department of Social and Economic Affairs. World Urbanization Prospects: The 2014 Revision Highlights. Published 2014. http://esa.un.org/unpd/wup/Highlights/WUP2014-Highlights.pdf. Accessed January 28, 2016.Google Scholar
5. Active Learning Network for Accountability and Performance in Humanitarian Action. The State of the Humanitarian System in 2015 Report. Published 2015. http://www.alnap.org/resource/21036.aspx. Accessed January 28, 2016.Google Scholar
6. Inter-Agency Standing Committee. IASC is the United Nations’ primary mechanism for interagency coordination of humanitarian assistance. https://interagencystandingcommittee.org/. Accessed, January 28, 2016.Google Scholar
7. World Health Organization. Health Emergency Risk Management Framework and Improving Public Health Preparedness. Published 2012. http://apps.who.int/iris/bitstream/ 10665/78359/1/WHO_HSE_HEA_HSP_2013.1_eng.pdf. Accessed January 28, 2016.Google Scholar
8. Cranmer, H, Chan, J, Kayden, S, et al. Pathway to professionalization: a competency-based evaluation of humanitarian aid workforce personnel during a humanitarian crisis simulation exercise. Prehosp Disaster Med. 2014;29(1).Google Scholar
9. Johnson, K, Idzerda, L, Baras, R, et al. Standardized training for humanitarian workers: the first step toward professionalization of the humanitarian community. Disaster Med Public Health Prep. 2013;7(4):369-372.Google Scholar
10. Enhancing Learning and Research for Humanitarian Assistance. Professionalizing the humanitarian sector: a scoping study. http://www.elrha.org/wp-content/uploads/ 2015/01/Professionalising_the_humanitarian_sector.pdf. Accessed January 4, 2016.Google Scholar
11. Enhancing Learning and Research for Humanitarian Assistance. Global Survey on Humanitarian Professionalization. Published 2014. http://euhap.eu/upload/2014/06/global-humanitarian- professionalisation-survey.pdf. Accessed January 4, 2016.Google Scholar
12. Okuda, Y. The utility of simulation in medical education: what is the evidence? Mt Sinai J Med. 2009;76(4):330.Google Scholar
13. Public Health Emergency Pre-Deployment Training in Tunisia. World Health Organization Regional Office for the Eastern Mediterranean. http://www.emro.who.int/surveillance-forecasting-response/surveillance-events/public-health-emergency-pre-deployment-training-in- the-region.html. Accessed January 28, 2016.Google Scholar
14. Humanitarian Response Intensive Course. Harvard Humanitarian Initiative. http://hhi.harvard.edu/education/workshops/hric. Accessed January 28, 2016.Google Scholar
15. Lateef, F. Simulation-based learning: just like the real thing. J Emerg Trauma Shock. 2010;3(4):348-352.Google Scholar
16. Canadian Disaster and Humanitarian Response Training Program. Humanitarian Studies Initiative, McGill University. https://www.mcgill.ca/familymed/global- health/courses/hsir/disaster-and-humanitarian-response-program. Accessed January 28, 2016.Google Scholar
17. Start Network. Core Humanitarian Competencies Framework. Published 2012. file:///Users/melaniecoutu/Downloads/CBHA-Core-%20Humanitarian-%20Competencies- Framework-colour.pdf. Accessed January 28, 2016.Google Scholar
18. Karpicke, JD, Blunt, JR. Retrieval practice produces more learning than elaborative studying with concept mapping. Science. 2011;331(6018):772-775.Google Scholar
19. World Health Organization. Health Action in Crises: Primary Health Care in Crises. Published 2009. http://www.who.int/hac/about/annual_report/annual_report_2008.pdf. Accessed January 28, 2016.Google Scholar
20. Savoldelli, GL, Naik, VN, Hamstra, SJ, Morgan, PJ. Barriers to use of simulation-based education. Can J Anaesth. 2005;52(9):944-950.Google Scholar
21. Welke, TM, LeBlanc, VR, Savoldelli, GL, et al. Personalized oral debriefing versus standardized multimedia instruction after patient crisis simulation. Anesth Analg. 2009;109(1):183-189.CrossRefGoogle ScholarPubMed
22. Larsen, DP, Butler, AC, Roediger, HL. 3rd, Test-enhanced learning in medical education. Med Educ. 2008;42(10):959-966.Google Scholar
23. Roediger, HL, Karpicke, JD. The power of testing memory. Perspectives on psychological. Science. 2006;1(3):181-210.Google Scholar
24. Roediger, HL, Karpicke, JD. Test-enhanced learning: taking memory tests improves long-term retention. Psychol Sci. 2006;17(3):249-255.Google Scholar
25. Bradt, DA, Drummond, CM. Professionalization of disaster medicine—an appraisal of criterion-referenced qualifications. Prehosp Disaster Med. 2007;22(5):360-368.Google Scholar
26. University of Laval program: Maîtrise en sciences de l’administration - gestion du développement international et de l’action humanitaire. https://www2.ulaval.ca/international/pole-dexcellence-sur-linternational/programmes-detudes/developpement-international-et-action-humanitaire.html. Accessed January 22, 2016.Google Scholar
27. McGill Humanitarian Studies Initiative (HSI). http://www.mcgill.ca/familymed/global-health/courses/hsir. Accessed January 22, 2016.Google Scholar
28. Humanitarian U Core Professional Humanitarian Training Program. www.humanitarianu.com. Accessed January 22, 2016.Google Scholar
29. Kamper, SJ, Maher, CG, Mackay, G. Global rating of change scales: a review of strengths and weaknesses and considerations for design. J Man Manip Ther. 2009;17(3):163-170.Google Scholar
30. Egleston, BL, Miller, SM, Meropol, NJ. The impact of misclassification due to survey response fatigue on estimation and identifiability of treatment effects. Stat Med. 2011;30(30):3560-3572.Google Scholar
31. Cosgrave, J, Ramalingam, B, Beck, T. Real-time evaluations of humanitarian action: an ALNAP guide. ALNAP. http://www.alnap.org/resource/5595. Accessed January 21, 2016.Google Scholar
32. Consortium of British Humanitarian Agencies. Evaluation of Consortium of British Humanitarian Agencies pilot. Published 2012. file:///C:/Users/ejohn/Downloads/1448.pdf. Accessed January 23, 2016.Google Scholar
33. Consortium of British Humanitarian Agencies. Context Humanitarian Staff Development Project Program Guide. http://www.contextproject.org/index.php. Accessed January 23, 2016.Google Scholar
34. Consortium of British Humanitarian Agencies. Humanitarian Capacity Building Program: Objective 1 Final Report. Published 2010. http://www.start-network.org/wp-content/uploads/2014/01/Humanitarian-Capacity-Building- Programme-%E2%80%93-Objective-1-Final-Report.pdf. Accessed January 23, 2016.Google Scholar
35. Johnson, K. Professionalizing Humanitarian Action. In: Caroline Abu-Sada, (ed). Dilemmas, Challenges, and Ethics of Humanitarian Action: Reflections on Médecins Sans Frontières’ Perception Project. Montreal, Quebec, Canada: McGill-Queen’s University Press; 2012: 104-115.CrossRefGoogle Scholar
36. Disaster Ready courses. www.disasterready.org. Accessed January 24, 2016.Google Scholar
37. LaRoche Global Development and Humanitarian Aid Training. http://www.laroche.edu/International_Outreach/Global_Development_and_Humanitarian_Aid_T raining/?gclid=CLS6q4_sgcsCFVRbhgodbEkNsw. Accessed January 24, 2016.Google Scholar
38. Lenchus, JD. End of the “See One, Do One, Teach One” era: the next generation of invasive bedside procedural instruction. J Am Osteopath Assoc. 2010;110(6):340-346.Google Scholar
39. Hasson, HM. Core competency in laparoendoscopic surgery. J Soc Lap Surg. 2006;10(1):16-20.Google Scholar
40. Accreditation Council for Graduate Medical Education. https://www.acgme.org/acgmeweb/. Accessed January 23, 2016.Google Scholar
41. McFadden, CL, Cobb, WS, Lokey, LS, Cull, DL, Smith, DE, Taylor, SM. The impact of a formal minimally invasive service on the residents’ ability to achieve new ACGME guidelines for laparoscopy. J Surg Educ. 2007;64(6):420-423.Google Scholar
42. Dulan, G, Rege, RV, Hogg, DC. Proficiency-based training for robotic surgery: construct validity, workload, and expert levels for nine inanimate exercises. Sure Endosc. 2012;26(6):1516-1521.Google Scholar
43. McGaghie, WC, Issenberg, B, Cohen, E, Jeffrey, B, Wayne, D. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011;86(6):706-711.Google Scholar
44. Ericsson, KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2005;79(10):70-81.Google Scholar
45. Eichbaum, Q. The problem with competencies in global health education. Acad Med. 2015;90(4):414-417.Google Scholar
46. Humanitarian Health Action: Emergency Medical Teams and World Health Organization. World Health Organization. http://www.who.int/hac/techguidance/ preparedness/emergency_medical_teams/en/. Accessed January 23, 2016.Google Scholar