SPODoM: search-based parameter optimization framework on just-in-time software defect prediction model

Abstract

Software is playing the most important role in recent industrial innovation, and consequently the amount of software has been rapidly growing last decades. For instance, safety-critical nature of vehicles makes software quality assurance (SQA) has become an essential prerequisite for such innovation. Just-in-time software defect prediction (JIT-SDP) is a special defect prediction method, which aims to conduct software defect prediction (SDP) on commit-level code changes for effective SQA resource allocation. JIT-SDP has advantages of fine granularity, automatic extraction, early application, and traceability. Recent research shows that JIT-SDP prediction model has still rooms for performance improvement since the hyperparameters of the machine learning model are not optimized yet according to characteristics of projects. Search-based software engineering is an approach to solve the problem as search problem formulated by search space and fitness function, e.g., Harmony Search (HS) is a widely used music-inspired meta-heuristic optimization algorithm. In this article, we propose search-based parameter optimization framework on JIT-SDP and demonstrate that our approach can produce the better performance of prediction and reduce effort in practice. Using 8 datasets from both industrial and open source software projects, we obtained an optimized model that meets the performance criterion beyond baseline of previous studies throughout various defect to non-defect class imbalance ratio of datasets. Experiments with open source software also showed better recall for all datasets despite we considered balance as performance index. Search-based parameter optimized JIT-SDP can be applied to the industrial domain software with high class imbalance ratio. We expect that our research can improve the performance of JIT-SDP even in both industrial software and open source software projects with different data characteristics. In addition, the cost-benefit analysis results showed that 20% effort enables the detection of 56% of defects on average and that the post-release quality cost can be reduced by 37.3% in practice. Finally, we also expect that our research can help reduce review effort and post-release quality costs.