Background subtraction (BS) is a crucial machine vision scheme for detecting moving objects in a scene. With the advent of smart cameras, the embedded implementation of BS finds ever-increasing applications. This paper presents a new BS scheme called efficient BS for smart cameras (EBSCam). EBSCam thresholds the change in the estimated background model, which suppresses variance of the estimates, resulting in competitive performance compared with standard BS schemes. The percentage of wrong classification of EBSCam is lower than those of the Gaussian mixture model (GMM) (10.97%) and the pixel-based adaptive segmenter (PBAS) (4.66%) algorithms in FPGA implementations. Moreover, the memory bandwidth requirement of EBSCam is 6.66%, 41.36%, and 90.48% lower than the state-of-the-art FPGA implementation of GMM, ViBe, and PBAS algorithms, respectively. EBSCam achieves a significant speed up compared with the FPGA implementations of GMM (by 43.3%), ViBe (by 118.6%), and PBAS (by 144.8%) schemes. Similarly, the energy consumption of EBSCam is 80.56% and 99.9% less compared with GMM and PBAS, respectively. In summary, the advantages of EBSCam in accuracy, speed, and energy consumption combined together make it especially suitable for embedded applications.