This paper presents a unified approach for the image understanding problem based on the MRF models. In the proposed scheme, the image segmentation and interpretation processes cooperate in the simultaneous optimization process so that the erroneous segmentation and misinterpretation can be compensately recovered by continuous estimation of the unified energy function.