Benchmarking Workflow Discovery: A Case Study From Bioinformatics

Automation in science is increasingly marked by the use of workflow technology. The sharing of workflows through repositories supports the verifiability, reproducibility and extensibility of computational experiments. However, the subsequent discovery of workflows remains a challenge, both from a sociological and technological viewpoint. Based on a survey with participants from 19 laboratories, we investigate the current practices in workflow sharing, re-use and discovery among life scientists chiefly using the Taverna workflow management system. To address their perceived lack of effective workflow discovery tools, we go on to develop benchmarks for the evaluation of discovery tools, drawing on a series of practical exercises. We demonstrate the value of the benchmarks on two tools: one using graph matching and the other relying on text clustering. Copyright (C) 2009 John Wiley & Sons, Ltd.