Seamless integration of soft wearable hand robots via augmented intelligence

Abstract

Hand function is a key factor for Activities of Daily Livings as people grab things to eat and drink with their hands. For some of people suffering from Spinal Cord Injury or stroke, their weak hand functions lead to difficulties in performing the activities. Soft wearable hand robots, due to their non-rigid material characteristics, have been recently developed to assist people with a loss of hand mobility. However, these robots are not yet used in real-life environments, only having been prototyped at the research level. This is because while there have been a lot of hardware developments of wearable hand robots, their intelligence is yet underdeveloped. This thesis addresses the intelligence for wearable hand robots to seamlessly augment hand functions. First, to acquire contact information while grabbing objects, a fingertip force estimation method is developed, which does not require any sensors attached to robots. This thesis proposes Bending Time-Gradient Long Short-Term Memory (BT-LSTM) that deals with the non-linearity and hysteresis of the system and dynamic changes in sheath bending angle. Second, a new intention detection paradigm proposed for the users wearing the robots based on the following hypothesis: user intentions can be inferred through observing 1. The moving trajectories of hand and 2. Hand-object interaction. This thesis proposes Vision-based Intention Detection Network from the EgOcentric view (VIDEO-Net), a deep learning model that detects user’s grasping and releasing intentions. Third, this thesis generalizes the hypothesis to enable users to perform various bimanual activities like preparing breakfast. It proposes a Bayesian approach, Proximal-distal, Action-Intention modeling from Vision (Pro-ActIV), to predict what users intend to do with their fingers by analyzing the kinematics of the hand’s proximal parts in relation to object locations. The results from the proposed approaches take a step forward in order to augment the ability of users with a loss of hand mobility to perform their intended tasks without causing discomfort.