The tendon-sheath mechanism (TSM) is applied to most flexible surgery robot systems because of its inherent advantages, such as small size, low weight, flexibility, and
efficient power transmission. However, several uncertainties, friction, and backlash can cause hysteresis, which interferes with control of a surgical instrument. In this paper, the image-based
hysteresis compensator is proposed. This compensator adjusts the position reference input using the visual-feedback concept. The proposed compensator utilizes learning-based pose
estimation using a siamese convolutional neural network (Siamese CNN). It estimates the instrument position without using markers. The proposed method is evaluated in a testbed
that is designed by considering the several requirements for an actual surgical instrument. As a result, we confirm that the image-based hysteresis compensator is able to decrease hysteresis. Moreover, the learning-based poses estimation works well without the use of markers. This result shows the feasibility of applying the proposed method to a real surgical robot system to facilitate hysteresis reduction.