(An) effective model combination approach based on adaptive criteria tree for software reliability prediction

Abstract

Accurate software reliability prediction during software testing is still an open challenge in reliability engineering communities. Although many software reliability models have been developed for this challenge, there is no single model appropriate in all circumstances. Accordingly, it is common to apply multiple software reliability models to the observed failure data, and some recent studies have focused on how to apply multiple models more effectively (e.g., model selection or combination based on some comparison criteria). However, it is not easy to identify the model that makes the most trustworthy long-and short-term reliability prediction. Thus, inappropriate model selection or combination often causes unsuccessful software reliability prediction in practice, which leads to cost/schedule overrun in a project. This dissertation aims to provide reliability practitioners with more accurate software reliability prediction from the mid stages of testing. As a preliminary step, a large-scale empirical study is conducted to investigate the characteristics of comparison criteria which have been commonly used in existing model selection or combination approaches. Study results show that a goodness-of-fit criterion to the past observations cannot properly capture the prediction performances of models. Such improper criteria application in existing approaches may cause the low accuracy in software reliability prediction. These motivated us to develop a new model combination approach. In this dissertation, we propose a model combination approach based on prediction confidence for more accurate software reliability prediction. To obtain the prediction confidence values of models, we developed Adaptive Criteria Tree (ACTree) which generates a series of criteria rules with consideration of multi-criteria joint effects. ACTree is constructed by identifying more informative criteria and their threshold values for reliability prediction, and the constructed tree assigns a prediction confidence value to each model. Then, the various combinations of models are examined with given prediction confidence values, and optimal one is used for the software reliability prediction. To evaluate the proposed approach, a series of experiments have been performed based on a leave-one-out cross validation technique. From experimental results, the proposed approach showed more stable and accurate prediction results than existing approaches in many cases regardless of prediction periods, which can help reliability practitioners to obtain more accurate software reliability metrics.