Cross-modal Correlation Analysis between Vowel Sounds and Color

Vowel-color association characteristics have been studied in the field of phonetics and perception. Though it has been reported that selected color categories after listening vowel categories have similar trends in multiple languages, their sentiment correlations have not yet been thoroughly studied from the viewpoint of speech features. We tried to find sentiment association characteristics between color parameters and speech features directly to have better understanding of cross-modal correlations and to find underlying principles for multimodal applications. Vowel samples uttered by 4 male and 3 female speakers were employed to associate colors after listening them by 34 subjects. Statistical analyses showed the advantage of employing RGB color parameters and speech formants directly to conventional color category to vowel category mapping. The selected color distributions in the F1- F2 plane clearly show that the acoustic speech resonance (i.e. F1 and F2) -RGB correlations can more consistently explain their sentiment correlations. Moreover, by incorporating our sentiment association experiment results using formant-synthesized speech, their correlations can be attributed to F1 and F2 rather than vowel categories. We believe that this finding in cross-modal correlations will serve for not only scientific understanding but also further studies and applications using cross-modal information mapping.