Validity of Using Mobile Phone Surveys to Evaluate Community Health Worker Program in Mali

Abstract BackgroundAssessing implementation strength through face-to-face interviews in hard-to-reach and unstable regions presents many challenges. Mobile phone-based interviews have become an alternative, but the validity of these data from this approach for Implementation Strength Assessment evaluation has not been sufficiently studied yet. The objective of this study was to assess the validity of mobile phone-based health provider interviews to measure the implementation strength of an integrated community case management (iCCM) and family planning program in Mali.MethodsFrom July to August 2018, interviewers administered a structured questionnaire to community health workers (ASCs) implementing the iCCM and family planning program in six districts in Mali. Interviews were conducted, first by phone, then verified through in-person visits. Survey questions addressed background information, training, supervision, demand generation activities and supplies of essential drugs and contraceptive methods. Sensitivity and specificity of the phone responses were calculated using the in-person response as the gold standard. A threshold of 80% for sensitivity and specificity respectively was considered acceptable.ResultsOf 157 ASCs interviewed by phone, 115 (73.2%) were reached in-person. Most indicators (9/10 iCCM indicators, 6/6 family planning indicators) for training, supervision, and availability of supplies on the day of interview, and those related to patient utilization reached the 80% threshold for sensitivity, but few (2/10 iCCM indicators, 1/6 family planning indicators) reached 80% for specificity. In contrast, most indicators of supply stock-outs in the last 3 months reached the threshold for specificity (5/6 iCCM indicators, 3/3 family planning indicators) but few reached the threshold for sensitivity (1/6 iCCM indicators, 1/3 family planning indicators).ConclusionsThe validity of data collected by phone were adequate for indicators of training, supervision, and day-of-interview commodity availability. Phone-based surveys are useful as a low-cost option for data collection in the assessment of implementation strength on general activities in inaccessible and resource-limited regions with mobile network connectivity.