Two-dimensional (2D) materials have recently provided a new perspective on optoelectronics because of their unique layered structure and excellent physical properties. However, their potential use as optoelectric devices has been limited by the trade-off between photoresponsivity and response time. Here, based on a vertically stacked atomically thin p-n junction, we propose a gap-mode plasmon structure that simultaneously enables enhanced responsivity and rapid photodetection. The atomically thin 2D materials act as a spacer for enhancing the gap-mode plasmons, and their short transit length in the vertical direction allows fast photocarrier transport. We demonstrate a high responsivity of up to 8.67 A/W with a high operation speed that exceeds 35 MHz under a 30 nW laser power. Spectral photocurrent, absorption, and a numerical simulation are used to verify the effectiveness of the gap-mode plasmons in the device. We believe that the design strategy proposed in this study can pave the way for a platform to overcome the trade-off between responsivity and response time.