Analyst and machine learning opinions in fire debris analysis

The principles of subjective logic are applied to the competing propositions that ignitable liquid residue (ILR) is present, or is not present, in a fire debris sample. Analysts' estimates of the strength of evidence coupled with their perceived levels of uncertainty combine to define a "fuzzy category" that is mapped to an opinion triangle. The opinion is expressed as a tuple consisting of the belief mass, disbelief mass, uncertainty and base rate. A workflow is introduced to guide the analyst through the fuzzy category formulation. Opinion tuples are also generated from a set of machine learning (ML) models trained on an ensemble of data sets. A set of 20 singleblind fire debris samples were analyzed by each of the authors, and by an ensemble of optimized support vector machine models. The opinions of each analyst and the ML ensemble were compared and combined to obtain an opinion representing a consensus of each analyst and the ML. The opinions of the analysts and ML were projected onto the zero-uncertainty axis and the projected opinion probabilities were used as scores to construct an receiver operating characteristic (ROC) curve. The area under the ROC curves for each analyst were greater than or equal to 0.90 and the area under the ML ROC curve was 0.96. The methodology is widely applicable to forensic problems that can be represented as a pair of mutually exclusive and exhaustive hypotheses.