Hardware/software co-design for efficient security analysis

Abstract

The reference monitor is one of the methodologies to enhance the security of a system. It defines a legitimate state of the system and determines anomaly by comparing runtime status with pre-defined legitimate status. One of the important factors to determine the performance and practicality of the monitor is the implementation method, which can be based on either software or hardware. Software-based approaches have the advantage that they can be easily applicable and deployable to the existing system without changing hardware components, but the monitoring code for checking the security of a protected system employes hardware resources (i.e., CPU and memory) occupied by the software running on the host, which causes performance degradation. Hardware-based approaches have a negligible performance overhead but they are not flexibly adopting new detection method, since any modification of an hardware would certainly incur significant costs and time. In this thesis, I will describe the hardware/software co-design for a practical security reference monitor that can be attached to an existing system without significant modification of a host processor. By splitting the monitoring into the information extraction and the security analysis, and offloading heavy security analysis to a dedicated off-core, the solutions can affect negligible performance overhead to the host system. Furthermore, the software components of the solutions provide not only the programmability and flexibility that security analysis schemes can be agilely applied but also the way the context information of the host processor is efficiently transferred to the off-core. Based on the above design, I implemented two security monitors based on the data flow and control flow analysis each of which can mitigate code-reuse attack and memory corruption vulnerabilities respectively.