Vision-based vibration measurement has the great advantage that the displacement response of a whole surface, not a point or a line, can be measured at one time. However, the measurement using a camera has a problem in that the sampling rate is insufficient as compared with the conventional contact type sensor. The main objective of this paper is to propose a vibration measurement method in the hyper-Nyquist frequency range using a vision system. We also apply this method to implement a new vision-based modal analysis system. For the experimental modal analysis, we propose a new measurement method using features that the experimental designer can select and use the input signal of the excitation part. The input signal phase can be obtained in advance through the Hilbert transform process for the excitation input signal. Based on this phase signal, a specific trigger signal can be designed to make measurements at the desired instantaneous phase. All data acquisition process for the modal testing is operated with this trigger signal. We implemented the proposed method in hardware using an FPGA (field-programmable gate array) board. The feasibility of the suggested method was performed through the vibration measurement experiment using a cantilever beam. As a simple experiment on the cantilever beam to validate the proposed system, vibration measurement and modal analysis of the hyper-Nyquist frequency range were performed.