Conventional endoscopes have difficulties in performing the endoscopic procedures for early cancer treatment due to limited movement. To overcome this, robotic platforms with steerable robotic surgical instruments have been developed. These platforms need to be smaller in size to enter the natural openings and need enough payload to handle organs. However, diameter and payload have a trade-off relationship and no clear study of this relationship has been preceded. In this paper, we presented a payload optimization of a flexible surgical instrument with rolling contact joint mechanism. The payload can be obtained from moment equilibrium equations on each rolling contact point, and the payload is optimized with constraints on required radius of curvature, minimum required diameter and wire slack protection. As a result, the payload increased about 40% more than the previously developed surgical instruments.