Constructing mobile autonomous agent with human-like app memory using LLM as a proactive task demonstrator

Abstract

The advent of large-scale language models (LLMs) has opened up new opportunities in mobile task automation. Thanks to their superior language comprehension and reasoning capabilities, users can automate complex and repetitive tasks. However, the inherent instability of LLMs significantly limit their practical applicability. To address these issues, this paper presents MobileGPT, an innovative LLM-based mobile task automation tool augmented by a unique app memory. MobileGPT emulates the cognitive processes by which humans interact with mobile apps to explore, select, derive, and recall. This approach makes learning more accurate and efficient by breaking down work procedures into small modular components that can be reused, rearranged, and adapted for different purposes. Additionally, its human-in-the-loop memory repair mechanism allows users to edit these modular components themselves, which reinforces the accuracy of MobileGPT. We implement MobileGPT using online LLM services (GPT-4). Throughout controlled experiments (N=25), we demonstrate the high usability of MobileGPT compared to other task automation tools. Furthermore, we found out the necessity of the repair mechanism in the task automator and our human-in-the-loop mechanism fulfilled that need with intuitive UI.