When does text boost images: an empirical investigation of the dynamics of information

Abstract

The contemporary surge in e-commerce activity has precipitated a complex digital marketplace teeming with both visual and textual content. This study addresses the intricate dynamics between these two modalities and their impacts on consumers' information processing and purchasing behaviors. Anchored in the burgeoning e-commerce domain, this research, with a particular focus on a major South Korean fashion e-commerce platform, parses a vast dataset encompassing over 15 million transactions and 705,056 images. Employing fixed effects regression analysis, the study investigates the influence of image and text information on sales volume. The transfer learning-based computer vision framework is designed to dissect the contributions of three image types, categorized based on object-level complexity. Additionally, utilizing the pre-trained language model, this study examines the extent of textual information to discern their individual and interactive effects on consumer purchase decisions. Furthermore, the research delves into the image-text interplay, examining the role of machine-extracted features in information processing, as guided by the Elaboration Likelihood Model (ELM). It categorizes textual information based on its redundancy with visual information and explores the differential effects of visual object-level complexity. These effects are contingent on product brand status and the degree of consumer involvement. This approach offers insights into the distinct ways consumers engage with and process multi-modal information in an e-commerce setting. The findings of this study contribute to a deeper understanding of digital consumer behavior, highlighting the pivotal role of multi-modal information processing in the online shopping landscape. This work elucidates the complex interrelations between visual and textual cues and provides a framework for e-commerce platforms to optimize content strategies, enhancing consumer engagement and sales.