The strong confinement of surface plasmons' optical fields at metal surfaces makes them highly sensitive to the structural shape and refractive index change of target biological(1,2), chemical(3,4) or atomic species(5). This has made surface plasmon resonance a widely applicable sensing technique. Plasmonic metrology is primarily based on the spectral shift of the scattering intensity spectrum. Although broadband phase spectra are known to provide richer information on target samples as opposed to intensity spectra, direct acquisition of broadband phase spectra in plasmonics has been made difficult by the lack of highly stabilized light sources. Here, we demonstrate that frequency-comb-referenced phase spectroscopy provides high speed, high resolution, and high linearity with respect to plasmonic rulers, with direct traceability to a time standard. As a demonstration, we measure the 1.94 angstrom dynamic motion of a pair of nanoholes with a resolution of 1.67 pm. The interaction through the propagation of the plasmonic field is enhanced by a factor of 155 compared to the physical sample length. Our realization of a fast and robust plasmonic ruler with picometre resolution makes it possible to obtain high-precision plasmonic phase spectroscopy for in-depth analysis of the dynamics of samples in nanoscopic volumes.