This paper presents our real-time human activity recognition system that understands human behavior using multimodal sensor data at multiple levels. Our system consists of a multimodal data acquisition framework and a user understanding algorithm including user identification, activity recognition, and health monitoring components.