GPUmore: Scalable Multi GPU Dataset Centric Network Servers

We present GPUmore, a programming model and a runtime for building scalable multi-GPU network servers for applications with large in-memory read-mostly datasets, like K-Nearest Neighbors search. Such services combine high compute intensity with huge datasets that by far exceed the limited GPU memory capacity. GPUmore abstracts the distributed memory architecture of multi-GPU systems by building a system-wide key-value store abstraction for accessing the application dataset, while dynamically caching the dataset in the memory of the GPUs. It introduces a novel fetch-and-execute processing primitive for explicit coupling between computations and data. Under the hood, GPUmore transparently optimizes the performance by caching the working set in the memory of the respective GPU, while balancing the load and improving access locality across GPUs via function migration and preemption, just like …