High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers.

The omission of nonlinear effects in large-scale 3D ground motion estimation, which are particularly challenging due to memory and scalability issues, can result in costly misguidance for structural design in earthquake-prone regions. We have implemented nonlinearity using a Drucker-Prager yield condition in AWP-ODC and further optimized the CUDA kernels to more efficiently utilize the GPU's memory bandwidth. The application has resulted in a significant increase in the model region and accuracy for state-of-the-art earthquake simulations in a realistic earth structure, which are now able to resolve the wavefield at frequencies relevant for the most vulnerable buildings (> 1 Hz) while maintaining the scalability and efficiency of the method. We successfully run the code on 4,200 Kepler K20X GPUs on NCSA Blue Waters and OLCF Titan to simulate a M 7.7 earthquake on the southern San Andreas fault with a spatial resolution of 25 m for frequencies up to 4 Hz.