P022: The revised METRIQ Score: an international, social-media based usability analysis of a quality evaluation instrument for medical education blogs

Abstract

Introduction: Online medical education resources are widely used in emergency medicine (EM), but strategies to assess quality remain elusive. We previously derived the Medical Education Translational Resources: Impact and Quality (METRIQ) 8 instrument to evaluate quality in medical education blog posts. Methods: As part of a subsequent validation study (The METRIQ Blog Study), a mixed-methods usability analysis was performed to obtain user feedback on the quality assessment instrument in order to improve its clarity and reliability. Participants in the METRIQ Study were first asked to rate five blog posts using the METRIQ-8 Score. They then evaluated the METRIQ-8 instruments ease of use and likelihood of being recommended to others using a 7-point Likert scale and free text comments. Participants were also asked to flag and comment on items within the score that they felt were unclear. Global usability ratings were summarized using median scores or percent rated unclear. We used ANOVA to test associations between ease of use and demographic factors. A thematic analysis was performed on the comments. Results: 309 EM medical students, residents, and attendings completed the survey. Global ratings were generally very favorable (median 2 [IQR 2-3], with 7 being the lowest score) for ease of use and likelihood of recommendation, and did not vary by participants country of origin, frequency of blog use, or learner level. Participants stated that the score was structured, systematic, and straightforward. They found it useful for junior learners and for guiding blog creation. Four questions in the score (questions 2, 4, 5, and 7) were identified by 10% of subjects to be unclear. Thematic analysis of comments identified suggested four main themes for improving the score: adding clearer definitions with marking rubrics; shortening the 7-point scale; adding items evaluating blog post presentation and utility; and, rephrasing the wording of certain questions for clarity. Conclusion: A mixed methods usability analysis of the METRIQ-8 instrument for assessing blog quality was globally well received by EM medical students, residents, and attendings. Qualitative analyses revealed multiple areas to improve the instruments clarity and usability. The METRIQ score is a promising instrument for evaluating the quality of blogs; further development and testing is needed to improve its utility.