Most background subtraction algorithms developed to detect moving objects are potentially problematic in that they experience performance degradation when weather conditions are adverse. We solve this problem by proposing a refinement method using a consistent motion detection method, the performance of which is robust to weather related changes in video images captured by a static camera. The proposed algorithm reduces the number of false-positive regions and fills parts that are missing as a result of the nature of the background subtraction methods. We show the extent of the improvement afforded by our algorithm in the handling of moving object detection in adverse weather conditions. (C) 2019 SPIE and IS&T