SHMT: Exploiting Simultaneous and Heterogeneous Parallelism in Accelerator-Rich Architectures

The addition of domain-specific hardware accelerators and general-purpose processors supporting vector and scalar models makes modern computers undoubtedly heterogeneous. However, existing programming models and runtime systems target using the most efficient category of processing units to delegate computation from each code region, undermining the potential parallelism that heterogeneous processing units can provide.

Simultaneous and heterogenous multithreading (SHMT) is a programming and execution model that activates all possible heterogeneous processing units for computation from a code region to enable “real” heterogeneous parallelism. SHMT presents an abstraction and a runtime system to facilitate parallel execution. Despite the new type of parallelism, SHMT also needs to additionally address the heterogeneity in data precision that various processing units support to ensure the quality of the result.

This paper implements and evaluates SHMT on an embedded system platform with a GPU and an Edge TPU. SHMT achieves up to 1.95× speedup and 51.0% energy reduction compared to GPU baseline.