(A) self-defensive approach to protect the software execution in the untrusted environment

Abstract

In the today``s most general-purpose computer system, the executable code, manipulated data, and program execution state are exposed to the execution environment. The execution environment can read, disassemble, or modify them as its own purpose. For example, cryptographic encryption - the most powerful method to conceal data must reveal the encryption key, decryption key, or plaintext to the execution machine. Once a program owner allows his or her program to run on an untrusted machine, there is usually no way to stop the owner of that machine from trying to spy on the program``s data, manipulate the program``s behavior, or damage the program in some way. However, in a modern security system, much attention and research has been focused on protecting user from malicious program that seek to steal data or do harm to the user``s machine. Unfortunately, however, not as much research has been focused on the reverse problem - that of protecting the executable code from attempts by malicious users to reverse engineer or modify the code for their own purposes. Malicious attacks can either come from not only external adversary, but also internal saboteur who is actually a legal user or participant. This is not so surprising since in the past, reverse engineering from native binary code has generally been a difficult task. As platform-independent and easily decompilable binary formats such as Java bytecode become more and more common today, however, one can no longer ignore this problem. There are many situations in which it is desirable to protect a piece of software from the illegitimate tampering once it gets distributed to the users. Protecting the software code means some level of assurance that the program will execute as expected even if it encounters some illegitimated modifications. In this dissertation, we present the methodology of protecting software from unauthorized modification. One important technique is an integrity-based encryption, by which a progra...