Continuous casting is the procedure of the successive casting for solidification of the steel, which contains several cooling processes along the caster to coagulate the molten steel. It is such a rule of thumb that strand surface quality and casting productivity is highly dependent on temperature control. A finite-difference method is one of estimating temperature distribution, yet it hinders the process control efficiently. Song, et al. suggest a multimodal deep learning approach for prediction of the temperature. However, sequential and transient phenomena of solidifying steel are not considered, which makes it difficult to estimate the sequential heat-transfer characteristics in the whole process of the steel concretion. Herein, a deep learning model is proposed to predict the temperature distribution by taking into account both transient and steady-state characteristics. The proposed model addresses both spatial and sequential information by incorporating a convolutional neural network (CNN) and a recurrent neural network (RNN). Our quantitative and qualitative results show considerable predictive performance improvement against baseline models. Furthermore, the proposed model is applicable in a real-world steel-making industry by providing real-time temperature prediction, whilst retaining a lower computational cost.