Measuring Knowledge of Community Health Workers at the Last Mile in Liberia: Feasibility and Results of Clinical Vignette Assessments

Introduction: Community health workers (CHWs) can provide lifesaving treatment for children in remote areas, but high-quality care is essential for effective delivery. Measuring the quality of community-based care in remote areas is logistically challenging. Clinical vignettes have been validated in facility settings as a proxy for competency. We assessed feasibility and effectiveness of clinical vignettes to measure CHW knowledge of integrated community case management (iCCM) in Liberia's national CHW program. Methods: We developed 3 vignettes to measure knowledge of iCCM illnesses (malaria, diarrhea, and pneumonia) in 4 main areas: assessment, diagnosis, treatment, and caregiver instructions. Trained nurse supervisors administered the vignettes to CHWs in 3 counties in rural Liberia as part of routine program supervision between January and May 2019, collected data on CHW knowledge using a standardized checklist tool, and provided feedback and coaching to CHWs in real time after vignette administration. Proportions of vignettes correctly managed, including illness classification, treatment, and referral where necessary, were calculated. We assessed feasibility, defined as the ability of clinical supervisors to administer the vignettes integrated into their routine activities once per year for each CHW, and effectiveness, defined as the ability of the vignettes to measure the primary outcomes of CHW knowledge of diagnosis and treatment including referrals. Results: We were able to integrate this assessment into routine supervision, facilitate real-time coaching, and collect data on iCCM knowledge among 155 CHWs through delivery of 465 vignettes. Diagnosis including severity was correct in 65%-82% of vignettes. CHWs correctly identified danger signs in 44%-50% of vignettes, correctly proposed referral to the facility in 63% of vignettes including danger signs, and chose correct lifesaving treatment in 23%-65% of vignettes. Both diagnosis and lifesaving treatment rates were highest for malaria and lowest for severe pneumonia. Conclusion: Administration of vignettes to assess knowledge of correct iCCM case management was feasible and effective in producing results in this setting. Proportions of correct diagnosis and lifesaving treatment varied, with high proportions for uncomplicated disease, but lower for more severe cases, with accurate recognition of danger signs posing a challenge. Future work includes validation of vignettes for use with CHWs through direct observation, strengthening supportive supervision, and program interventions to address identified knowledge gaps.