ARM TrustZone provides a Trusted Execution Environment (TEE) to isolate security-critical services, which are generally invoked from the Rich Execution Environment (REE) through a communication channel established by executing the Secure Monitor Call (SMC) with the general registers configured as input parameters. Unfortunately, the communication channel has been abused by adversaries to incur misbehavior of the TEE, to analyze the internal working of the TEE, and to exploit its vulnerabilities. We therefore propose the TEE defense (TFence) framework that enables the creation of a partially privileged (par-priv) process, which benefits from the coordination of the system mode and virtualization extension. More specifically, on ARM architecture, direct invocation of hypercall and SMC is not allowed in the user process; however, we limitedly escalate the privilege of the process to enable it to directly communicate with trust anchors such as hypervisor and TrustZone. This approach enabled us to remove the kernel dependency when the process communicates with the TEE, which also reduces the attack surface to the critical part of the application involved in the communication. Besides, direct communication with the hypervisor facilitates the adoption of application-shielding approaches to protect the critical part and to restrict arbitrary access to the TEE.