Presented is a robust visual lock-on framework for an unmanned aerial vehicle (UAV) that utilises geometric relations between the UAV pose and the 3D local map defined by the positions of the target and natural landmarks. Experimental results using real datasets demonstrate the robustness of the proposed method compared to state-of-the-art visual tracking methods.