Automated mutation-adequate test generation for function block diagram programs

Abstract

Function block diagram (FBD) is a standard programming language for programmable logic controllers (PLCs). PLCs have been widely used to develop safety-critical systems such as nuclear reactor protection systems. It is crucial to test FBD programs for such systems effectively. Research mainly focused on automated FBD test generation to achieve certain structural coverage criteria. Tests achieving a high coverage level can detect errors but fail to provide the assurance of fault detection. Mutation testing is an error-based testing technique to detect specific types of errors. To give some assurance of fault detection, this paper presents an automated test sequence generation approach using mutation testing techniques for FBD programs and the developed tool, MuFBDTester. MuFBDTester translates the given program and mutants into the input language of a satisfiability modulo theories (SMT) solver to derive a set of test sequences. The primary objective is to find the test data that can distinguish between the results of the given program and mutants. We conducted experiments with several examples including real industrial cases to evaluate the effectiveness and efficiency of our approach. The results indicate that the mutation-based test suites identified a tremendously high percentage of the artificial faults, including the faults that test suites generated to meet coverage criteria could not identify. The test suite generation time of MuFBDTester was significantly reduced compared to manual generation. When compared to coverage-based test generation time, it was sometimes longer, but reasonable considering its better fault detection effectiveness. MuFBDTester can provide highly effective test suites for FBD engineers.