Drum sample retrieval from mixed audio via a joint embedding space of mixed and single audio samples

Abstract

As the development of digital audio processing has popularized the technology of making music easily, sample-based music creation has become a mainstream practice. One of the key tasks in the sample-based approach is to search desired instrument samples in the large collections. However, most commercial sample packages described the samples using metadata, making it difficult to intuitively imagine the sound without listening to it. Inspired by music producers who often find instrument samples with a reference song, we set up a query-by-example scheme that takes mixed audio as a query and retrieves single audio samples. Our method is based on deep metric learning where a triplet neural network is trained to have single audio samples and their mixtures with other instruments closely located in the embedding space. We also suggest a method to generate mixed audio to build the dataset. As a result, we observe the performance difference according to the learning method, the model configuration, and the learning input types to find the best model for retrieving single audio in mixed audio. The results show that our model achieves promising retrieval performance in the query-by-example task. We also ensure the operation of the neural network by visualizing both single and mixed audio samples in the embedding space.