The validity and reliability of the digital Ruff Figural Fluency Test (RFFT) in the general adult population: Epidemiology / Innovative methods in epidemiology (i.e., assessment methods, design, recruitment strategies, statistical methods, etc.)

Background The Ruff Figural Fluency Test (RFFT) is a valid pencil‐and‐paper test for evaluating cognitive functioning in the general population. The assessment of the pencil‐and‐paper‐RFFT is time‐consuming and requires training of raters. We developed a digital‐RFFT which can be filled in by people independently on an Ipad, after which RFFT patterns are automatically processed and stored. We aimed to investigate the validity and reliability of the digital‐RFFT in the general population. Method Stratified randomization (based on age, gender, and education) was used to allocate participants either to the digital‐RFFT or the pencil‐and‐paper‐RFFT. After the first test, the other test was repeated (cross‐over). The number of unique designs and perseverative errors of the pencil‐and‐paper‐RFFT and digital‐RFFT were assessed by an computerized algorithm (automatic) and by two independent trained human raters (criterion standard). For the statistical analysis, we used Intraclass correlation coefficients(ICC), sensitivity, specificity, %agreement, weighted Kappa, and Bland Altman plots, including limits‐of‐agreement(LoA). Results The study population comprised 96 participants (58% women; mean(SD) age 40(15)years). The mean(SD) number of unique designs was 84.2 (26.0) (automatic assessment) and 85.3 (26.2) (human assessment) for the digital‐RFFT and 91.3 (22.7) for the pencil‐and‐paper‐RFFT. The median number(IQR) of perseverative errors was 4(2‐7) (automatic assessment) and 4.5(2‐8) (human assessment) for the digital‐RFFT and 4.5(2‐7.5) for the pencil‐and‐paper‐RFFT. The percentage agreement (kappa) between the two human raters for the digital‐RFFT and pencil‐and‐paper‐RFFT was 0.94(0.90) and 0.94(0.81), respectively. ICC between human assessment and automatic assessment of the digital‐RFFT (unique designs) was 0.988(95%CI:0.981‐0.992); sensitivity and specificity was 0.98 and 0.96 respectively; mean difference(95%CI):‐1.12(‐1.92,‐0.33); 95%Limits‐of‐Agreement ‐8.75,6.51). The ICC between digital‐RFFT and pencil‐and‐paper‐RFFT was 0.53(95%CI 0.37,0.67). When starting with the paper‐and‐pencil‐RFFT, this ICC was 0.73 (95%CI:0.34,0.87). The mean difference (unique designs) between the digital‐RFFT and pencil‐and‐paper‐RFFT was ‐7.09 (95%CI ‐11.8 to ‐2.38 (95%limits‐of‐agreement: ‐52.12, 37.94). This was due to unclear instructions (i.e. complex examples) that were given on the Ipad (digital‐RFFT) and learning effect. Conclusions Although small and systematic differences between human assessment and automatic assessment of the digital‐RFFT were found, we conclude that the digital‐RFFT is a valid test to assess cognitive functioning among the general population if instructions of the digital‐RFFT are improved.