We present three-dimensional (3-D) target tracking based on fused radar and infrared (IR) sensor data with the inclusion of target orientation in the measurement vector. We provide the noise statistic of IR-sensor measurements, including target orientation measured from the IR image. The track-to-track fusion with extended Kalman filter is used to combine radar with IR sensor data. In conventional tracking approaches, there is a fundamental limitation in that it is difficult to accurately estimate the current acceleration of the target, even with nearly perfect measurements of range and angle relative to the target. The correlation between target orientation and velocity can be used to overcome this limitation. We evaluate tracking performance to show how much improvement is obtainable through the inclusion of the target orientation in the measurement data for a realistic 3-D scenario.