Tracking objects using unmanned aerial vehicles (UAVs) has been widely utilized in various fields. However, target tracking in a cluttered environment can be challenging because of the unexpected target movement and ambiguous surroundings. In this paper, a fast and robust target tracking system for UAVs is proposed to tackle these problems. A simple network-based detection module and a filter-based object tracking module are utilized and developed not to miss the target even in occlusion or among similar vehicles. The upcoming path of the target is predicted using the traces of a target, and the model predictive controller is utilized to track the nonuniform movement as they are. Moreover, the yaw compensator module is designed to track the target robustly to minimize the noise and react fast to the agile target motion. The performance of the proposed system is verified by tracking the target in challenging urban simulation environments. (https://youtu.be/pMfhb25DqDU)