An Intelligence Architecture for Grounded Language Communication with Field Robots

Abstract

For humans and robots to effectively collaborate as teammates in unstructured environments, robots must be able to construct semantically rich models of the environment, communicate efficiently with teammates, and perform sequences of tasks robustly with minimal human intervention as direct human guidance may be infrequent and/or intermittent. Contemporary architectures for human-robot interac- tion often rely on engineered human interface devices or structured languages that require extensive prior training and inherently limit the information that humans and robots can communicate. Natural lan- guage, particularly when situated with a visual representation of the robot’s environment, allows humans and robots to quickly and effectively exchange information about abstract goals, specific actions, and/or properties of the environment in addition to serving as a mechanism to resolve inconsistencies in the mental models of the environment across the human-robot team. This article details a novel intelligence architecture that leverages a centralized representation of the environment informed by a visual per- ception pipeline, declarative knowledge, deliberative interactive estimation, and a multi-modal interface to perform complex tasks in unstructured environments. The language pipeline also exploits proactive symbol grounding to resolve uncertainty in ambiguous statements through inverse semantics. A series of experiments on three different unmanned ground vehicles demonstrates the utility of this architecture through its ability to robustly perform language-guided spatial navigation, mobile manipulation, and bidirectional communication with human operators. Experimental results give examples of component- level behaviors and overall system performance that guide a discussion on observed performance and opportunities for future innovation.