Automated unit test generation with realistic unit context synthesis for low false alarms

Abstract

Current testing practice in industry is often ineffective and inefficient to detect bugs since most test cases are created manually. In addition, the execution space of a complex target program is too large to explore in limited testing time. As a solution for these problems, automated unit test techniques automatically generate drivers/stubs for each unit of a target program and test cases to explore the execution space of each target unit separately. However, these techniques suffer a large number of false alarms due to approximated inaccurate unit contexts which allow infeasible executions of a target unit.

In this dissertation, I present an automated unit test generation framework that synthesizes realistic unit context to automatically detect bugs with low false alarms. The first part of this dissertation presents CONBOL which is the world's first automated unit testing framework for large industrial C programs. CONBOL generates symbolic unit testing drivers/stubs automatically and applies heuristics to reduce false alarms caused by the imprecise drivers/stubs. CONBOL demonstrated its bug detection effectiveness by detecting 24 new crash bugs in a four million lines long industrial embedded program. The second part of this dissertation presents the world's most accurate automated unit testing framework CONCERT which reduces a large number of false alarms by automatically synthesizing realistic unit contexts. CONCERT synthesizes test drivers/stubs to represent realistic contexts of a target unit f by utilizing the code of the other units which are closely relevant to f. The relevance of other unit g to f is measured based on how many times g and f are executed together in system executions. In the experiments on the 67 crash bugs of the 15 real-world C programs (55KLOC on average), CONCERT demonstrates both high bug detection ability (i.e., 83.6% of the target bugs detected) and low false/true alarm ratio (i.e., 2.4 false alarms per one true alarm).

As future work, I will improve the function correlation metric for reducing false alarms further. Also, I plan to develop a framework to build system test cases based on automatically generated unit test cases. Furthermore, I will utilize unit relevance information for other purposes such as impact analysis.