One of the challenging issues in TV recommendation applications based on implicit rating data is how to make robust recommendation for the users who irregularly watch TV programs and for the users who have their time-varying preferences on watching TV programs. To achieve the robust recommendation for such users, it is important to capture dynamic behaviors of user preference on watched TV programs over time. In this paper, we propose a topic tracking based dynamic user model (TDUM) that extends the previous multi-scale dynamic topic model (MDTM) by incorporating topic-tracking into dynamic user modeling. In the proposed TDUM, the prior of the current user preference is estimated as a weighted combination of the previously learned preferences of a TV user in multi-time spans where the optimal weight set is found in the sense of the evidence maximization of the Bayesian probability. So, the proposed TDUM supports the dynamics of public users' preferences on TV programs for collaborative filtering based TV program recommendation and the highly ranked TV programs by similar watching taste user group (topic) can be traced with the same topic labels epoch by epoch. We also propose a rank model for TV program recommendation. In order to verify the effectiveness of the proposed TDUM and rank model, we use a real data set of the TV programs watched by 1,999 TV users for 7 months. The experiment results demonstrate that the proposed TDUM outperforms the Latent Dirichlet Allocation (LDA) model and the MDTM in log-likelihood for the topic modeling performance, and also shows its superiority compared to LDA, MDTM and Bayesian Personalized Rank Matrix Factorization (BPRMF) for TV program recommendation performance in terms of top-N precision-recall.