Trusted execution environment for multi-level security and hardware sandbox

Abstract

Recent studies showed that a cloud application consists of multiple distributed modules provided by mutually distrustful parties. Although hardware-based trusted execution environments (TEEs) have evolved to provide strong isolation with efficient hardware supports their current monolithic model poses challenges in representing the application consisted of multiple modules produced by mutually distrustful parties. For trusted services, such applications can use TEEs that are communicating through software-encrypted memory channels. For better mapping of such modular software designs to trusted execution environments, this dissertation proposes a hierarchical TEE model called Nested Enclave, and a bi-directional TEE model called Cloister. Nested Enclave extends the current monolithic model to a hierarchical one, which provides multiple inner TEEs within a TEE. Inspired by the multi-level security model, nested enclave provides multiple inner enclaves sharing the same outer enclave. Inner enclaves can access the context of the outer enclave, but they are protected from the outer enclave and non-enclave execution. Peer inner enclaves are isolated from each other while accessing the execution environment of the shared outer enclave. Both inner and outer enclaves are protected from vulnerable privileged software and physical attacks. Such fine-grained nested enclaves allow secure multi-tiered environments using software modules from untrusted 3rd parties. The security-sensitive modules run on the inner enclave with the higher security level, while the 3rd party modules are on the outer enclave. It can be further extended to provide a separate inner module for each user to process privacy-sensitive data while sharing the same library with efficient hardware-protected communication channels. This study investigates three case scenarios implemented with an emulated nested enclave support, proving the feasibility and security improvement of the nested enclave model. Cloister provides a new type of bi-directional protection: protecting the rest of the system from the enclave module with sandboxing and protecting the enclave module from third-party modules and the operating system. However, the current TEE model cannot efficiently represent such distributed sandbox applications. To overcome the lack of hardware support, Cloister proposes new three key techniques. First, it extends the hardware-based memory isolation in SGX to confine a user software module only within its TEE (enclave). Second, it proposes a trusted monitor enclave that filters and validates system calls from enclaves. Third, it allows hardware-protected memory sharing between a pair of enclaves for efficient protected communication without software-based encryption. Finally, it facilitates a mutually agreeable accounting system among cloud providers and users. Using an emulated SGX platform with the proposed extensions, this study shows that distributed sandbox applications can be effectively supported with small changes of SGX hardware.