This letter proposes a monitoring system to prevent falls from height accidents at construction sites. In our previous work, a method was proposed for recognizing the fastening state of the safety hook based on the motion similarity between two inertial measurement unit sensors, but its performance was limited to 90.64% Youden's Index (YI). This study introduces a safety hook monitoring system that achieves better performances by utilizing a deep Siamese neural network-based model to develop valid feature representations, which enhance performances. Our proposed approach achieves 97.69% YI, surpassing previous works in recognition performances.