Visuo-haptic information is essential for efficient object perception. Here we investigated the neural response to congruent and incongruent visuo-haptic object stimuli using fMRI. Compared to incongruent stimuli, congruent visuo-haptic stimuli evoked bigger response in early sensory cortex as well as traditional multisensory area such as lateral occipital cortex (LOC) and posterior parietal cortex (PPC). The congruency effect in the early sensory cortex contrasts with the traditional view that multisensory processing occurs at the high level association cortex where information from different primary sensory cortex converges. However, a recent hierarchical inference model of the brain provides an elegant explanation for the involvement of sensory-specific cortex as well as high level cortex. Precision or gain of early sensory channels can be larger for congruent multisensory input because the channels are regarded as more reliable sources that facilitate the inference process. Furthermore, higher multi-voxel pattern reproducibility was observed for the congruent stimuli and it also implies more reliable inference on the sensory cause.