Perception-to-planning network for episodic memory of home service robots in cooperation with human

Abstract

To perform a home service task through cooperation with a human in a real environment, a robot needs to deal with the environmental changes and accordingly plan appropriate behavior sequence. For this purpose, in this paper, we propose a Human-Robot Collaboration Architecture which is based on memory and reasoning. A robot perceives user behaviors and objects using an RGB-depth and thermal sensor. The robot stores a temporal sequence of behaviors for performing a task in its episodic memory, Perception-to-Planning Network, that is realized by a sequence to sequence network. When the user command is given, the episodic memory is used to retrieve the behavior sequence to carry out the command. On the other hand, when the robot perceives user behaviors, the robot postpones its behavior till his/her behavior is stopped. Once stopped, the episodic memory retrieves the behavior sequence to conduct a task that the user has intended. The internal simulator schedules the behavior sequence from the memory and confirms the behavior sequence to be executable and then if executable, it sends the next executable behavior to the execution module. If a behavior fails in the internal simulation test, fast forward planner generates an alternative behavior sequence to resolve the failed behavior problem. The effectiveness and applicability of the proposed Perception-to-Planning Network and Human-Robot Collaboration Architecture are demonstrated by a wheel-based humanoid robot. In addition, we have demonstrated the possibility of implementing a cooking recommancation robotic system.