Remote monitoring of implementation fidelity using log-file data from multiple online learning platforms

The impact of educational programs on student learning is contingent upon the quality and fidelity of their implementations. Yet, the most reliable method of implementation monitoring, direct observation, may be infeasible for large-scale efficacy studies. Here, we use log-file data from an efficacy study assessing the impact of gamified learning technologies on algebraic knowledge to measure the fidelity of implementations remotely within and across multiple programs. We employ exploratory-data-analysis to identify measures of fidelity, multilevel models to evaluate which levels of implementation contexts (i.e. classrooms, teachers, students) were associated with variance in fidelity metrics, and latent profile analysis to identify high and low-fidelity contexts. These analyses illustrate how log-file data may be used to remotely monitor and evaluate the implementations of online learning platforms.