Time perception is crucial for human episodic memory. Previous studies frequently reported the activation of the temporal perception in the human prefrontal cortex (PFC), the hippocampus, and the parietal lobe. However, the specific subregions involved in episodic temporal perception in these areas have not been revealed yet. In this study, we proposed that the PFC and the parietal lobe, and the hippocampus are activated by scene-based episodic temporal order memory. To test the hypothesis, we performed an event-related functional magnetic resonance imaging (fMRI) experiment. During the scan, cortical activities were recorded while participants memorized and retrieved the temporal order of visual scene stimuli. We found that inferior frontal gyrus (IFG), middle frontal gyrus (MFG), and the superior frontal gyrus (SFG) were significantly activated during the retrieval of the temporal order. The activity of MFG and SFG had a significant negative correlation with the temporal distance. As the results of multivariate pattern analysis, MFG, SFG showed great accuracy. Therefore, the results suggest that the temporal order judgment, one of essential components in episodic memory, is represented in MFG and SFG.