Critical thinking about treatment effects in Eastern Africa: development and Rasch analysis of an assessment tool

Background: Every day we are faced with different treatment claims, in the news, in social media, and by our family and friends. Some of these claims are true, but many are unsubstantiated. Without being supported by reliable evidence such guidance can lead to waste and harmful health choices. The Informed Health Choices (IHC) Network facilitates development of interventions for teaching children and adults the ability to assess treatment claims (informedhealthchoices.org). Our objective was to develop and evaluate a new assessment tool developed from the item bank for use in an upcoming trial of lower secondary school resources in Uganda, Kenya, and Rwanda. Methods: A cross-sectional study evaluating a questionnaire including two item-sets was used. The first evaluated ability using multiple-choice questions (scored dichotomously) and the other evaluated intended behaviour and self-efficacy (measured using Likert scales). This study was conducted in Uganda, Kenya, and Rwanda in 2021. We recruited children (over 12 years old) and adults through schools and our networks. We entered 1,671 responses into our analysis. Summary and individual fit to the Rasch model (including Cronbach’s Alpha) were assessed using the RUMM2030 software. Results: Both item-sets were found to have good fit to the Rasch model and were acceptable to our target audience. The reliability was good (Cronbach’s alpha >0.7). Observations of the individual item and person fit provided us with guidance on how we could improve the design, scoring, and administration of the two item-sets. There was no local dependency in either of the item-sets, and both item-sets were found to have acceptable unidimensionality. Conclusion: To our knowledge, this is the first instrument validated for measuring ability to assess treatment claims in Uganda, Kenya and Rwanda. Overall, the two item-sets were found to have satisfactory measurement properties.