Modulation of the refractive index of materials is elementary, yet it is crucial for the manipulation of electromagnetic waves. Relying on the inherent properties of natural materials, it has been a long-standing challenge in device engineering to increase the index-modulation contrast. Here, we demonstrate a significant amount of ultrafast index modulation by optically exciting non-equilibrium Dirac fermions in the graphene layer integrated onto a high-index metamaterial. Furthermore, an extremely-large electrical modulation of refractive index up to Delta n similar to -3.4 (at 0.69 THz) is achieved by electrical tuning of the density of the equilibrium Dirac fermion in the graphene metamaterial. This manifestation, otherwise remaining elusive in conventional semiconductor devices, fully exploits the characteristic ultrafast charge relaxation in graphene as well as the strong capacitive response of the metamaterial, both of which enable us to drastically increase the light-matter interaction of graphene and the corresponding index contrast in the graphene metamaterials.