On the effectiveness of same-domain memory deduplication

ABSTRACTMemory deduplication, an OS memory optimization technique that merges identical pages into a single Copy-on-Write (CoW) page, has been shown to be susceptible to a variety of timing side channel attacks, all of which stem from the differences between write times to the CoW page and to the normal page. To mitigate this issue, operating systems only merge pages from the same security domain (e.g., from the same process); moreover, browsers can piggyback on this defense with the recent adoption of site isolation. This was all considered sufficient, because it thwarts existing attacks, which have all relied upon separate domain (e.g., cross-process) scenarios. In this paper, we examine the effectiveness of same-domain memory deduplication as a mitigation by presenting two case studies that show that an attacker can still leverage the deduplication side channel to leak secrets. Specifically, our case studies highlight one key flaw: that it is non-trivial to separate programs into separate security domains. In the first case study, we examine a client-server scenario---a scenario that inherently requires a server to read data from an untrusted client---and demonstrate that the client can control the alignment of data in memory to disclose the server's secret data. In the second case study, we examine a recent version of Firefox---a browser that has undergone massive efforts to ensure that data from different origins are separated into different domains---and demonstrate that nonetheless, a malicious webpage can exploit the browser's partial implementation of site isolation to leak secret data across tabs. We conclude that same-domain memory deduplication as a defense is difficult to implement correctly, and hence, is insufficient.