This study describes an experimental study on dynamic characteristics improvement of a helicopter hingeless rotor system in the viewpoint of aeroelastic stability improvement and low vibration. To do this work, a baseline hingeless hub system and blade were developed. For the study of aeroelastic stability improvement, a baseline hingeless hub system was developed and composite materials have been applied to improve the aeroelastic stability and to reduce the weight of the rotor hub system. For the study of a low-vibration blade, a baseline rotor blade was developed and composite materials have been applied to a well-tailored design focused on optimization of the blade sectional property distribution. In this experimental study, some parts of a baseline hub system were replaced with composite parts to improve the aeroelastic stability. In addition to this aeroelastic improvement of the rotor system, a weight reduction effect was also achieved of about 56%. To verify this aeroelastic stability improvement, hover and wind-tunnel tests were performed. An improvement of the aeroelastic stability was achieved at about 30%. In addition, a low-vibration small-scale blade was developed by the applying a composite material tailoring method to adjust the mass distribution along the blade. A small-scale blade was designed and tested to verify the low-vibratory characteristics comparison with an existing baseline blade. The improvement of vibration reduction was noted to be about 10-20%.