A new algorithm and its hardware architecture are presented to up-scale high-definition (HD) and full-HD video streams to 4-K ultra-HD video streams in real time. The Lagrange interpolation is employed, as it provides high estimation accuracy and hardware-friendly properties. To enhance the accuracy further, the pixels at the edge regions are specially processed by employing an image-sharpening technique. Experimental results show that the proposed architecture provides the best visual quality at the cost of reasonable hardware resources.