Sim-to-Real Transfer of Image-Based Autonomous Guidewire Navigation Trained by Deep Deterministic Policy Gradient with Behavior Cloning for Fast Learning

Abstract

Percutaneous coronary intervention (PCI) is a frequently used surgical treatment for cardiovascular disease, one of the leading cause of death in the world. In traditional PCI, a doctor navigates a thin guidewire in a patient's vessel toward a target location by looking into live X-ray angiogram images of the patient. Recently, researchers are using reinforcement learning to automate this guidewire navigation process without attaching any sensor to the guidewire tip. These researchers use a real vessel phantom to train their behavior policy using reinforcement learning. Training a reinforcement learning algorithm on a real setup can give a good guidewire control on that setup, but it is under question whether the trained algorithm can be applied to other vessel structures. We can make various vessel phantoms and train the algorithm on the setups, but it can be really time and money consuming. In this paper, we devise a method for sim-to-real transfer of a guidewire navigation trained by reinforcement learning using only images. We pretrain our behavior policy using data collected by running an expert algorithm in the virtual environment. Then, we train the behavior policy by deep deterministic policy gradient (DDPG) in a virtual environment. With behavior cloning, our method learns to successfully navigate a guidewire in much shorter time than training DDPG from scratch without behavior cloning. After done with the training, we transfer the behavior policy trained in the virtual environment to the guidewire navigation in a real vessel phantom. Our trained behavior policy navigates the guidewire to destinations successfully in all test episodes and navigates faster than the expert algorithm. Experiment video is available at: https: //youtu.be/HCEbIhZsXqw