By directly measuring electrical hysteresis loops using the Positive-Up Negative-Down (PUND) method, we determined accurately the remanent ferroelectric polarization P-r of orthorhombic RMnO3 (R = Ho, Tm, Yb, and Lu) compounds below their E-type spin ordering temperatures. We found that LuMnO3 has the largest P-r of 0.17 mu C/cm(2) at 6 K in the series, the value of which allows us to predict that its single-crystal form can produce a P-r of at least 0.6 mu C/cm(2) at 0 K. Furthermore, at a fixed temperature, P-r decreases systematically with increasing rare earth ion radius from R = Lu to Ho, exhibiting a strong correlation with the variation of the in-plane Mn-O-Mn bond angle and Mn-O distances. Our experimental results suggest that the contribution of the Mn t(2g) orbitals may dominate the ferroelectric polarization.