Scalable Wide-Area Resource Discovery

This paper describes the design and implementation of SWORD, a scalable resource discovery service for wide-area distributed systems. SWORD locates a set of machines matching user-specified constraints on both static and dy- namic node characteristics, including both single-node and inter-node characteristics. We explore a range of system ar- chitectures to determine the appropriate tradeoffs for build- ing a scalable, highly-available, and efficient resource discov- ery infrastructure. We describe: i) techniques for efficient handling of multi-attribute range queries that describe ap- plication resource requirements; ii) an integrated mechanism for scalably measuring and querying inter-node attributes without requiring O(n2) time and space; iii) a mechanism for users to encode a restricted form of utility function in- dicating how the system should filter candidate nodes when more are available than the user needs, and an optimizer that performs this node selection based on per-node and inter-node characteristics; and iv) working prototypes of a variety of architectural alternatives—running the gamut from centralized to fully distributed—along with a detailed performance evaluation. SWORD is currently deployed as a continuously-running service on PlanetLab. We find that SWORD offers good performance, scalability, and robust- ness in both an emulated environment and a real-world de- ployment.