Directed Energy Deposition (DED) is a metal additive manufacturing technology that is gaining popularity for its ability to rapidly manufacture virtually any metal components no matter how complex the shapes and properties are. However, the current lack of real-time geometry monitoring and control is hindering the wider dissemination of DED in industries. This study developed and validated a geometry monitoring methodology which can achieve real-time inspection of the melt pool and newly solidified layer, and layer-wise inspection of the deposited layer during DED process. An encoder-decoder network was developed and applied to the profile images from the laser line scanner to obtain track profiles. A point cloud generation method was proposed to convert the obtained track profiles into 3D point cloud data using intrinsic/extrinsic calibration and printing position. Experiments have been successfully conducted to validate the proposed methodology by depositing multi-layer X-shape objects.