Empirical evaluation on FBD model-based test coverage criteria using mutation analysis

Abstract

Function Block Diagram (FBD) is a graphical modeling language for Programmable Logic Controller (PLC) programs. Recently, FBD has been increasingly used to implement safety-critical software such as nuclear reactor protection software. With increased importance of structural testing for FBD models, FBD model-based test coverage criteria have been already introduced: Basic Coverage (BC), Input Condition Coverage (ICC), and Complex Condition Coverage (CCC). While the three FBD test coverage criteria reflect data flow-characteristics of FBD and suggest intuitive structural coverage concepts, fault detection ability of the FBD coverage criteria is not evaluated.

In this paper, we empirically investigate the effectiveness, and strength/weakness of the fault detection of the FBD test coverage criteria. Also, we analyze the gain of using the coverage criteria in terms of the test effectiveness and cost by comparing to the random testing. To gain statistically significant results, we generate 1500 test suites satisfying the FBD test coverage criteria. Meanwhile, we generate 481 faults of FBD models by applying mutation operators to five industrial FBD models. Then we investigate which test suite detects what mutants.

In summary, the strongest FBD test coverage criterion, CCC, can detect at most 94.4\% of faults in the subject model, and the average fault detection ability of BC and ICC were not significantly different in most cases. Though the three coverage criteria have strength on detecting faults in Boolean edges and arithmetic blocks, faults in constant values, logic blocks, and comparison blocks are not detected effectively. These findings improve empirical background of the structural testing for FBD models.