Auditory cortex neural response-inspired sound event detection

Abstract

Sound event detection is a task of recognizing sound event class and its corresponding onset and offset timestamps. It is fundamental and crucial task for auditory perception that auditory system can perform the task naturally. Organs inside auditory system process incoming sound in numerous temporal and spectral scale. Those processed information from preceding organs converges to auditory cortex and it is known to play a critical role in auditory perception. In this work, I applied auditory cortex neural response inspired method to sound event detection. Recently, performance of sound event detection has shown leap-up performance due to deep learning methods which demonstrate notable strength in pattern recognition. However, deep learning methods utilizes data leveraging methods and large models that lack not only interpretability but also understanding of acoustic and auditory domain. Therefore, the objective of this research is performance advancement and retaining interpretability by applying auditory cortex neural response inspired method. Spectro-temporal receptive field is utilized for imitation of auditory cortex neural response. In auditory neuroscience domain, spectro-temporal receptive field is used for prediction of auditory neural response for arbitrary sound which comes into ears. Spectro-temporal receptive field represents auditory neural response characteristic and auditory neural response can be predicted by convolution of arbitrary sound spectrogram and spectro-temporal receptive field along time axis. Auditory cortex neuron responses actively to certain spectro-temporal modulation that spectro-temporal receptive field of auditory cortex reflects the spectro-temporal modulation selectivity property. Based on response characteristic of auditory cortex, I constructed idealized spectro-temporal receptive fields and utilize them as filters of convolutional layer in deep learning model. However, constructed filters have large shape that detailed time-frequency information cannot be captured through constructed spectro-temporal receptive fields. To tackle the limitation, I built two-branch structure that one branch captures various spectrotemporal modulation through spectro-temporal receptive field, and another branch captures detailed time-frequency information through normal convolutional layer kernels. Additionally, I showed effectiveness of spectro-temporal receptive field on bioacoustic event detection task. A bioacoustic event retains plentiful spectro-temporal modulation that spectro-temporal receptive field would help to capture those event. A bioacoustic event detection is set to few-shot learning task that it is important to extract class representation given few examples. Spectro-temporal receptive field is fixed filter that it has further advantage to extract the information efficiently from few examples, showing higher performance than other models. Lastly, I applied another human auditory system inspired feature which is cochleagram replacing melspectrogram. A performance using both feature shows similar trends and proposed model performs better than other models at both task and also when using both input feature which are melspectrogram and cochleagram. By so, I showed the effectiveness of extracting spectro-temporal modulation information by auditory cortex inspired method.