DynamicLabels: Supporting informed construction of machine learning label sets with crowd feedback

Abstract

Label set construction --- deciding on a group of distinct labels --- is an essential stage in building a machine learning (ML) application, as a badly designed label set negatively affects other stages such as training dataset construction, model training, and model deployment. Despite its significance, it is challenging for ML practitioners to come up with a well-defined label set, especially when no external references are available. To mitigate this difficulty, ML practitioners often go through multiple iterations to gradually improve their label set. Through our formative study (n=4), we observed that there still remain challenges in collecting helpful feedback and utilizing them to make optimal refinement decisions. To support the iterative refinement, we present DynamicLabels, a system that aims to support a more informed label set-building process with crowd feedback. Crowd workers provide feedback as annotations and label suggestions to the ML practitioner's label set, and the ML practitioner can review the feedback through multi-aspect analysis and see the potential consequences of label refinements. Through a within-subjects study (n=16) using two datasets, we found that DynamicLabels enables better understanding and exploration of the collected feedback and supports a more structured, confident refinement process. The ML practitioners were also able to see surfacing conflicts and edge cases that could have been ignored. In addition, the crowd feedback helped ML practitioners to gain diverse perspectives, spot current weaknesses, and shop from crowd-generated labels. With DynamicLabels, ML practitioners can successfully gain concrete understanding and evidence from the crowd and make informed refinements to iteratively improve the label set.