An implementation of MPI and hybrid OpenMP/MPI parallelization strategies for an implicit 3D DDG solver

This work describes an implementation of OpenMP, MPI, hybrid OpenMP/MPI parallelization strategies for an implicit three-dimensional (3D) direct discontinuous Galerkin (DDG) solver used for Navier–Stokes equations. Significantly, an efficient local matrix-based MPI parallelization strategy of an implicit DG solver is proposed. Furthermore, we give an implementation of OpenMP and hybrid OpenMP/MPI strategy for comparison. The storage structure based on the local matrix makes the MPI parallelization can easily use the local number to access and assign, the storage is compact and compatible with Block Sparse Row (BSR) format, and the program is easier to modularize. Several numerical tests for 3D Navier–Stokes equations are implemented to indicate the performance of parallelization strategies. For the problem of more than 200 million degrees of freedom, the designed pure MPI strategy for 3rd-order DDG solvers with 2nd-order polynomials (DDG(P2)) can get parallel efficiency of almost 90% at near ten thousand cores on the Tianhe-2 supercomputer. In particular, the pure MPI parallelization based on local matrix reaches a higher level of parallel efficiency than hybrid OpenMP/MPI parallelization.