A robot capable of proactive assistance through handovers for sequential tasks

Abstract

For robots to coexist with humans in diverse situations, their ability to fluently interact with humans becomes important. One important aspect of interacting with humans is being able to understand what the humans are doing to provide appropriate forms of assistance. Previous works used information from hands and objects to understand the human behavior and its context. However, as environments, tasks, and interaction targets become more complex, it becomes difficult to design assistance rules that can cover the variety of situations with such simple reasoning methods. Therefore, we develop a robotic system that combines action recognition with an activity-level knowledge bank to assist a human performing a sequential activity. The system maps the detected action to objects related to the task using the knowledge bank and delivers the objects to the human through handover. To evaluate the performance of our system, we conduct comparative experiments with two other simple systems: command-initiated and random-trial. Through experiments on two cooking tasks, our system is compared to the two simple systems on the basis of human idle time and object idle time. Results show that our system leads to the shortest human idle time. The object idle time of our robot system is similar to the command-initiated system and much shorter than the random-trial system. We conclude that robots that understand human actions can more efficiently assist humans to accomplish their tasks. © 2021 IEEE.