Calibration of few-shot classification tasks

Abstract

As numerous meta-learning algorithms improve performance when solving few-shot classification problems for practical applications, accurate prediction of uncertainty has been considered essential. In meta-training, the algorithm treats all the generated tasks equally and updates the model to perform well on the training tasks. For the training, some of the tasks might be hard for the model to infer the query examples from the support examples, especially when a huge mismatch between the support set and the query set exists. The distribution mismatch makes the model have wrong confidences that cause a calibration problem. In this study, we propose a novel meta-training method that measures the distribution mismatch and lets the model predict with more careful confidence. Moreover, our method is algorithm-agnostic and readily expanded to include a range of meta-learning models. Through extensive experiments, including dataset shift, we present that our training strategy helps the model avoid being indiscriminately confident, and thereby, produce calibrated classification results without the loss of accuracy.