Inprobability model based rate control of video coding, modeling of residual distribution is important in predicting precise distortions so as to determine appropriate quantization parameter values. For this, single probability model approaches have been popularly taken which may fail to model the underlying statistical characteristics of different residues from variable block-sized coding. In this letter, new rate and distortion models based on a mixture of multiple Laplacian distributions are presented for the transform coefficients of inter-predicted residues in quadtree coding. The proposed mixture model of multiple Laplacian distributions is tested for the High Efficiency Video Coding (HEVC) Test Model (HM) with quadtree-structured Coding Unit and Transform Unit. The experimental results show that the proposed model achieves more accurate results of rate and distortion estimation than the single probability models.