On developing a realtime learning-based control framework for network resource management

Abstract

As network environments become more complex and user demands become more diverse, the limitations of rule-based network management techniques are becoming apparent. Therefore, learning-based network resource management techniques that can learn optimal policies without human intervention have been actively researched in recent years. However, under time-varying network environments, policies learned in an offline manner with limited environments cannot guarantee optimal performance. In this study, we investigate realtime learning-based control for network resource management that can learn the given network environment quickly and data-efficiently with minimal performance loss. Specifically, we propose a realtime learning-based control framework for the downlink scheduling algorithm and a realtime learning-based control framework for the congestion control technique. We first present a reinforcement learning-based network scheduling algorithm for a single-hop downlink scenario that achieves throughput optimality and converges to minimal delay. To this end, we first formulate the network optimization problem as a Markov decision process (MDP) problem. Then, we introduce a new state-action value function called $Q^+$-function and develop a reinforcement learning algorithm called $Q^+$-learning with UCB (Upper Confidence Bound) exploration which guarantees small performance loss during a learning process. We also derive an upper bound of the sample complexity in our algorithm, which is more efficient than the best known bound from Q-learning with UCB exploration by a factor of $\gamma^2$ where $\gamma$ is the discount factor of the MDP problem. Furthermore, we propose a novel realtime learning-based control framework for downlink scheduling under time-varying environments. We first transform a problem with the objective of optimal throughput and queueing delay under time-varying environments into a piece-wise non-stationary MDP problem and design modules needed to enable real-time learning with the transformed problem. Specifically, we propose a novel method to leverage prior experiences in order to learn new optimal policies more efficiently compared to the random exploration algorithm. Finally, we consider a realtime learning-based control framework for congestion control. Though ideas of exploiting learning-based controls have been tried, none of them has succeeded in realizing such an ideal control due to the following fundamental challenges: 1) While time-varying network states require a learning-based control to keep learning the environment and optimal actions, it is unknown how to let it learn the optimal action without performing poorly during the learning process, and 2) it is under-explored how to identify and classify such time-varying states that have not been encountered before (i.e., unseen environments) in a detailed manner, which is crucial for learning the environment continually. To provide an answer to these challenges, in this work, we propose a new learning-based congestion control, namely CLINE, with the following two techniques: 1) CLINE predicts best-projected actions under an unseen environment by exploiting and extrapolating its inductive bias on the mapping structure between current observable network states and optimal states, which is learned during its offline training process with a finite set of environments and further improved by an online calibration process. 2) CLINE identifies and classifies the given network much more precisely by utilizing packet timing information and makes it possible for the mapping structure to expand by accumulating the experiences in each unseen environment over time.