Musical word embedding for natural language based music annotation and retrieval

Abstract

With the growth of online music streaming services, understanding user contexts has emerged as a significant issue. The users' context is expressed in various languages, and the language is used as a query for the music search system. Existing supervised learning-based music classification tasks result in the inability to respond to users' context queries due to finite-size labels. Accordingly, in the field of recent music information retrieval, using a word embedding that expresses the relative meaning between languages, both audio and language are mapped into an joint embedding space through zero-shot learning. However, word embeddings used are not specific to the music domain, and there is a problem that does not reflect the proper noun of music domains such as artists and song titles. In this paper, we propose musical word embeddings specialized in musical domains rather than word embeddings with a general meaning and propose similarity searches using proper nouns of musical domains. We also apply it to multi-label zero-shot learning using various musical information (tags, artists, and tracks) from musical word embeddings. We propose an architecture and learning method that leverages a joint loss function of music information that is very effective in music annotation and retrieval tasks and outperforms. We also show that the proposed architecture and learning methods are also useful for knowledge transfer. This means that the joint embedding space, combined with audio and language, reflects various music information while retaining generalization performance.