Long-term episodic and semantic memories with application to home service robot-IoT

Abstract

Automated home referred as Smart Home is expected to offer fully customized services to the residents, reducing the amount of home labor thus improving the welfare for human beings. Service robots and internet of things (IoT) play the key roles in the development of Smart Home. Service provision with these two main components in a Smart Home environment requires 1) learning and reasoning algorithms and 2) the integration of robot and IoT systems. Conventional computational intelligence based learning and reasoning algorithms do not successfully manage dynamic changes in the Smart Home data, and the simple integrations fail to fully draw the synergies from the collaboration of the two systems. To tackle these limitations, we propose 1) a stabilized memory network with a feedback mechanism which can learn user behaviors in an incremental manner and 2) a robot-IoT service provision framework for a Smart Home which utilizes the proposed memory architecture as a learning and reasoning module and exploits synergies between robot and IoT systems. We conduct a set of comprehensive experiments under various conditions to verify the performance of the proposed memory architecture and the service provision framework and analyze the experiment results. Intelligent agents need to gather relevant information and perceive semantics within the environments the agents are situated in before taking on given tasks. The agents store the collected information in the form of environment models, which represent the surrounding environments in a compact way. The agents, however, can only conduct limited tasks without an efficient and effective environment model. Thus, such a model of surrounding environments takes a crucial role in autonomy systems of intelligent agents. We claim the following characteristics for a versatile environment model: accuracy, applicability, usability, and scalability. Although a number of researches have attempted to develop such representations of environments and represented environments precisely to a certain degree, they lack broad applicability, intuitive usability, and satisfactory scalability. To tackle these limitations, we propose 3D scene graph as an environment model and the 3D scene graph construction framework. The concise and widely-used graph structure readily guarantees usability as well as scalability for 3D scene graph. We demonstrate the accuracy and applicability of 3D scene graph by exhibiting deployment of 3D scene graph in practical applications. Moreover, we verify the performance of the proposed 3D scene graph and the framework by conducting a series of comprehensive experiments under various conditions. Intelligent agents need to understand the surrounding environment to provide meaningful services to or interact intelligently with humans. The agents should perceive geometric features as well as semantic entities inherent in the environment. Contemporary methods in general provide one type of information regarding the environment at a time, making it difficult to conduct high-level tasks. Moreover, running two types of methods and associating two resultant information requires a lot of computation and complicates the software architecture. To overcome these limitations, we propose a neural architecture that simultaneously performs both geometric and semantic tasks in a single thread: simultaneous visual odometry, object detection, and instance segmentation (SimVODIS). Training SimVODIS requires unlabeled video sequences and the photometric consistency between input image frames generates self-supervision signals. The performance of SimVODIS outperforms or matches the state-of-the-art performance in pose estimation, depth map prediction, object detection, and instance segmentation tasks while completing all the tasks in a single thread. We expect SimVODIS would enhance the autonomy of intelligent agents and let the agents provide effective services to humans.