On the performance of spoken language translation in multimedia communication environment

Abstract

This dissertation is concerned with the performance of spoken language translation in multimedia communication environment. Current spoken language translation systems suffer not only from limited performance of each component module, but also from the accumulation of errors throughout the serial connection of components. Among the components, the speech recognition module is a major bottleneck and its performance is much lower than acceptable. We propose a spoken language translation system which provides its users with the multimedia aid for understanding the intention of a foreign speaker. Our approach to achieve high end-to-end, i.e., human-to-human performance in contrast to those of most conventional speech translation systems pursuing only the high input-to-output performance. The proposed system can provide even partially recognized utterances to the user, in order not to neglect the possibility that the user can infer the intended message using the context and his intelligence. Our system also provides a multimedia channel which can be utilized in the successful understanding of the error-prone translated speech. The multimedia channel is also used to reduce the errors from the out-of-vocabulary problem. We also propose a performance measure for spoken language translation systems. Our measure is defined as the ratio of the information quantities at each end of communication. Using this measure, we show that multimedia information can enhance spoken language translation systems. The experiments of Korean-to-English and Korean-to-Japanese spoken language translation systems show that the proposed measure is coherent with human understandability of the speaker``s intention.