With the proliferation of cloud computing, security concerns about confidentiality violations of user data by the privileged domain and system administrators have been growing. This paper proposes secure cloud architecture with a hardware security module, which isolates cloud user data from potentially malicious privileged domains or cloud administrators. Within a securely isolated execution environment, the hardware security module provides essential security functionality with only restricted interfaces exposed to vulnerable management systems or cloud administrators. Such restriction prevents cloud administrators from affecting the security of guest VMs. The proposed architecture not only defends against wide attack vectors but also achieves a small TCB. This paper discusses our hardware and software implementation of the proposed cloud architecture, analyzes its security, and presents its performance results.