Data Mining for Efficient Collaborative Information Discovery

The cybersecurity community expends considerable effort on establishing protocols, data formats, and coordination centers for sharing operational security information. There is widespread agreement that sharing information should create value, but also that it is far from simple for one organization to use intelligence provided to it by another. Substantial work focuses on engineering ontologies and data formats to resolve syntactic, and to some extent semantic, differences. These solutions aim to create high quality low noise shared data resources, but require substantial commitments in technology, man hours, and inter-organizational relationship building. Such expenditures may be beyond the reach of many organizations, especially since a substantial portion of the resulting shared data will remain unused. We contend that applying data mining and statistical learning methods to more easily obtainable, inconsistently or entirely unstructured data can guide and prioritize effort. We demonstrate these ideas with a case study of the incident reports collected by US-CERT in the course of one year. We find that data mining techniques can identify subsets of the indicator and incident landscapes for which the exchange of complete incident information may be useful to analysts and decision makers. The techniques studied here may allow broader participation in information sharing efforts, and make better use of the valuable resources dedicated to collaborative cybersecurity information discovery.