With the increasingly frequent appearance of mobile phones in college classrooms, there have been growing concerns regarding their negative aspects including distractive off-task multitasking. In this work, we design and evaluate Let’s FOCUS, a software-based intervention service that assists college students in self-regulating their mobile phone use in classrooms. Our preliminary survey study (with 47 professors and 283 students) reveals that it is critical to encourage voluntary participation by framing intervention as a learning tool and to raise awareness regarding appropriate mobile phone usage by establishing social norms in colleges. Let’s FOCUS introduces a virtual limiting space for each class (or a virtual classroom) where the students can explicitly restrict their mobile phone use voluntarily. Furthermore, it promotes students’ willing participation by leveraging social facilitation and context-aware reminders associated with virtual classrooms. We conducted a campus-wide campaign for approximately six weeks to evaluate the feasibility of the proposed approach. The results confirm that 379 students used the app to limit 9,335 hours of mobile phone usage over 233 classrooms. Let’s FOCUS was used in diverse learning contexts and for different purposes and its social learning and context-awareness features significantly motivated prolonged participation. We present the design considerations of software-based intervention.