Web-Scale Data Integration: You can afford to Pay as You Go

The World Wide Web is witnessing an increase in the amount of structured content - vast heterogeneous collections of structured data are on the rise due to the Deep Web, annotation schemes like Flickr, and sites like Google Base. While this phenomenon is cre- ating an opportunity for structured data management, dealing with heterogeneity on the web-scale presents many new challenges. In this paper, we highlight these challenges in two scenarios - the Deep Web and Google Base. We contend that traditional data in- tegration techniques are no longer valid in the face of such hetero- geneity and scale. We propose a new data integration architecture, PAYGO, which is inspired by the concept of dataspaces and em- phasizes pay-as-you-go data management as means for achieving web-scale data integration.