Analysis and modeling of array redistributions

Array redistribution is an indispensable tool for data-parallel languages like HPF. The availabilty of redistribution, often selected through compiler directives, gives the programmer freedom to select the best algorithm and associated data distribution for each task in a program. Redistribution is an expensive operation, however, and the programmer would like to ensure that the time saved after a redistribution is more than the cost of the redistribution itself. No comprehensive performance models have been developed to describe the cost of a redistribution, which can be divided into three phases: packing data, communication, and unpacking data. Here we eschew the traditional approach, which is to model only the communication costs, and instead perform a communication-independent analysis of the cost of packing and unpacking data. The cost of these operations can dominate the total redistribution cost. Our results show that an accurate model of the packing and unpacking costs for a variety of redistributions can be build. An accurate model must take into account hardware parameters such as Translation Lookaside Buffer (TLB) size, TLB replacement policy, TLB miss cost, cache black size, and cache miss cost. We show that cache collisions dominate the model for unpacking received data into a local array in a strided fashion. The number of collisions is a function of the redistribution performed, hence the programmer may not be able to eliminate them. The show how widely applicable our results are, we validate our model on three different CPU platforms.