In recent years, soft robotic grippers have emerged as a promising solution for versatile and safe manipulation of objects in various fields. However, precise force control is critical, especially when handling delicate or fragile objects, to avoid excessive grip force application or to prevent object slippage. Herein, we propose a novel three-degree-of-freedom force sensor incorporated within a soft robotic gripper to realize stable grasping with force feedback. The proposed optical sensor employs lightweight and compact optical fibers, thereby allowing for cost-effective fabrication, and a robust sensing system that is immune to electromagnetic fields. By innervating the soft gripper with optical fibers, a durable system is achieved with the fibers functioning as a strengthening layer, thereby eliminating the need for embedding an external stiffening structure for efficient bending actuation. The innovative contact-based light loss sensing mechanism allows for a robust and stable sensing mechanism with low drift (<0.1% over 9000 cycles) that can be applied to soft pneumatic bending grippers. We used the developed sensor-incorporated soft gripper to grasp various objects, including magnetic materials, and achieved slip detection along with grip force feedback without any signal interference. Overall, this study proposes a robust measuring multi-degree-of-freedom force sensor that can be incorporated into grippers for improved grasping stability.