We use the Vlasov-Maxwell equations to investigate the kinetic instability caused by the interaction of the electron beam with the electromagnetic wave in small signal limit. The orthogonality condition of the electromagnetic modes has been used to derive the dispersion relation for the dielectric-loaded cylindrical waveguide driven by a relativistic electron beam when a constant guiding magnetic field is applied. When the guiding field is not so strong, there arises cyclotron Cherenkov instability as well as Cherenkov instability. We show that the cyclotron Cherenkov instability has a growth rate comparable to that of the Cherenkov instability, and the amplification of an electromagnetic wave is possible in the range of millimeter to sub-millimeter arising from the anomalous Doppler effect.