Advanced mobile operating system for transparent multi-device interaction

Abstract

Today, with the advancement of mobile and IoT technologies, we live in multi-device environments consisting of not only smartphones but also various types of smart devices such as watches, cars, refrigerators and so on. With this trend, interests in how to use multiple devices together have been growing, and several use cases are emerging to show that simultaneous and cooperative use of multiple devices can provide new and useful user experience (UX). For example, when a user wants to play a sensor-based racing game on a smart TV, he can enjoy more comfortable game control by borrowing sensors of his smartphone. As another example, in the case of live-streaming apps, users have often encountered inconvenient situations in which a keyboard window covers a broadcast window when chatting with other viewers while watching the broadcast. If the keyboard window is migrated to another device, we can enjoy live streaming more conveniently than before. However, since the existing mobile operating systems are not designed for multi-device environments, the current multi-device usage is achieved through the app-level approach that employs some custom apps, which gives rise to several problems. First, from the perspective of app developers, substantial development effort for multi-device apps is required. Particularly, the development difficulties and error incidence necessarily increase as the range of target multi-device scenarios broadens and the cross-device communication becomes complicated. On the other hand, from the perspective of users, the potential of multi-device environments may be extremely limited, because we cannot employ many legacy apps that are released previously while utilizing resources of multiple devices. In addition, even when using custom apps, there are bothersome burdens that users should manually handle app installations and device configurations. These points can seriously hinder the popularization of multi-device environments. This dissertation proposes a novel system-level abstraction that makes apps on different devices run as if they were placed on the same single-device boundary. It can narrow the design gap between the current mobile operating systems and multi-device environments while addressing the above problems fundamentally. In the first detailed study, we designed Mobile Plus, a mobile operating system that allows apps on different devices to share their functionalities with each other by extending inter-app interaction to multi-device environments. Basically, on a single device, apps interact with each other using inter-process communication (IPC) for functionality sharing. The abstraction provided by Mobile Plus enables apps on multiple devices to interact with each other as if they are placed on the same single device, so legacy apps can share their functionalities in multi-device environments without any modification to them. We show that our system can support 22 useful multi-device scenarios using 20 popular legacy apps. In the second detailed study, we designed FLUID, a mobile operating system that can utilize surfaces (i.e., touch screens) of multiple devices in a flexible manner by extending intra-app interaction to multi-device environments. Our system can distribute user interface (UI) elements of apps to other devices to allow users to control multiple surfaces as they want. After that, the abstraction provided by FLUID enables UI elements and app logic on different devices to interact with each other as if they are on the same process boundary. It allows legacy apps to utilize multiple surfaces without modifying their source codes. We show that our system can support 10 useful multi-device scenarios with 20 popular legacy apps.