Interfacial perpendicular magnetic anisotropy (PMA) in CoFeB/MgO structures was investigated and found to be critically relied on underlayer material and annealing temperature. With Ta or Hf underlayer, clear PMA is observed in as-deposited samples while no PMA was shown in those with Pt or Pd. This may be attributed to smaller saturation magnetization of the films with Ta or Hf underlayer, which makes the PMA of CoFeB/MgO interface dominates over demagnetization field. On the contrary, samples with Pt or Pd demonstrate PMA only after annealing, which might be due to the CoPt (or CoPd) alloy formation that enhances PMA.