This paper presents the design of an optical torque sensor that can structurally decouple the effect of axial-interference for use in various robotics applications. Torque sensors are widely used in the joints of intelligent service or wearable robots to realize safe human-robot interaction. Whole robot body sensing using torque sensors is essential for safe interaction. However, most torque sensors are bulky, heavy and expensive. Therefore, many optical-based torque sensors have been proposed to deal with such problems, but the issue of axial-interference still remains. We resolved the axial-interference problem via the geometrical structure of the sensor body and differential signaling using two reflective optical sensors. The moment interference error was successfully decreased from 4.49% (with one optical sensor) to 0.11% (with two optical sensors) using the proposed sensor structure while maintaining a compact size, lightweight, and low cost. Static tests and dynamic tests were carried out and analyzed for accuracy error, hysteresis, and repeatability. We then compared the performance of an impedance controller that is widely used in service and wearable robots using the proposed sensor and a commercial torque sensor with respect to various control loop rates. The control performance of the proposed sensor was comparable to that of commercial sensors.