Direct numerical simulations (DNSs) are performed to investigate the roughness effects on the statistical properties and the large-scale coherent structures in the turbulent channel flow over three-dimensional sinusoidal rough walls. The outer-layer similarities of mean streamwise velocity and Reynolds stresses are examined by systematically varying the roughness Reynolds number k(+) and the ratio of the roughness height to the half-channel height k/d. The energy transfer mechanism of turbulent motions in the presence of roughness elements with different sizes is explored through spectral analysis of the transport equation of the two-point velocity correlation and the scale-energy path display of the generalized Kolmogorov equation. The results show that, with increasing k(+), the downward shift of the mean streamwise velocity profile in the logarithmic region increases and the peak intensities of turbulent Reynolds stresses decrease. At an intermediate Reynolds number (Re-t = 1080), the length scale and intensity of the large-scale coherent structures increase for a small roughness (k(+) = 10), which leads to failure of the outer-layer similarity in rough-wall turbulence, and decrease for a large roughness (k(+) = 60), as compared with the smooth-wall case. The existence of the small roughness (k(+) = 10) enhances the mechanism of inverse energy cascade from the inner-layer small-scale structures to the outer-layer large-scale structures. Correspondingly, the self-sustaining processes of the outer-layer large-scale coherent structures, including turbulent production, interscale transport, pressure transport and spatial turbulent transport, are all enhanced, whereas the large roughness (k(+) = 60) weakens the energy transfer between the inner and outer regions.