Memory vulnerabilities defense with object layout randomization

Abstract

This dissertation introduces randomization methods, which are one of the ways to protect existing memory vulnerabilities. Among these randomization methods, we point out a problem with randomizing the layout of an object that randomly changes the layout of a variable of an object. We propose a complementary method of changing the layout of the variable randomly each time when an object is allocated. This method solves the problem of the previous one, which is that the layout can be recovered in binary. However, since the method requires additional arithmetic operations to access object variables, it caused a performance problem. In this dissertation, we also present a new methodology for selecting objects to be protected using taint analysis and a hardware accelerator to improve performance as a solution for a performance problem. By utilizing taint analysis to find objects that attackers can control, we reduced the performance slowdown factor by protecting only exploitable objects. In addition, Hardware accelerator improved the performance by accelerating the arithmetic operations of the defense system.