Analyzing Dialog Coherence Using Transition Patterns in Lexical and Semantic Features

In this paper, we present methods to analyze dialog co- herence that help us to automatically distinguish be- tween coherent and incoherent conversations. We build a machine learning classifier using local transition pat- terns that span over adjacent dialog turns and encode lexical as well as semantic information in dialogs. We evaluate our algorithm on the Switchboard dialog cor- pus by treating original Switchboard dialogs as our co- herent (positive) examples. Incoherent (negative) exam- ples are created by randomly shuffling turns from these Switchboard dialogs. Results are very promising with the accuracy of 89% (over 50% baseline) when inco- herent dialogs show both random order as well as ran- dom content (topics), and 68% when incoherent dialogs are random ordered but on-topic. We also present ex- periments on a newspaper text corpus and compare our findings on the two datasets.