We consider globally coupled random frequency oscillators under thermal noise, and explore the synchronization transition with its critical behavior near the transition. In particular, we focus on the finite-size scaling behavior of the synchronization, and investigate how the thermal noise affects the correlation size exponent (v) over bar of the synchronized oscillators. Extensive numerical simulations as well as mean-field analysis have been performed. We find that the correlation size exponent changes from (v) over bar = 5/2 without thermal noise to (v) over bar = 2 with strong thermal noise, where the value (v) over bar = 2 is the same as that for the usual equilibrium systems described by the Ginzburg-Landau mean-field theory. In order to see the effects of thermal fluctuation further, we remove the frequency-disorder fluctuations originating from the different realizations of natural frequencies of the oscillators, and examine the finite-size scaling behavior for the case only with the thermal fluctuation. It is found that (v) over bar becomes 2 at much weak thermal noise strength, which implies that even very weak thermal fluctuations may lead to (v) over bar = 2 when frequency-disorder fluctuations are absent.