Recently, mobile traffic is increasing exponentially due to the development of personal devices such as smart phones and tablets. Bandwidth-hungry applications such as streaming data and multimedia file sharing are approaching 50% of the total traffic volume in 4G network.  In addition, due to the emergence of ultra-high bandwidth consuming applications such as virtual reality and augmented reality, the traffic volume is expected to increase more than 500 times in 5G. In order to solve these problems, various technologies have been proposed in 5G. As one of the candidate technologies for increasing the network capacity, a Small Cell Network (SCN), where a number of Small cell Base Stations (SBSs) are deployed per unit area by reducing the size of a cell, is attracting attention.
An SBS is a base station having a range of several tens to several hundreds of meters using a small power. Currently, the gap between cells in the mobile communication network is getting smaller within several hundred meters in the hotspot area, and it is expected to be constantly increased.  However, as the size of the cell decreases, inter-cell interference problem, backhaul link cost problem for the small cell deployment, and frequent cell changes due to the movement of a mobile user are promoting many research challenges on the SCN. In addition, interest in proactive caching in the SCN has increased significantly with the evolution of 5G technologies such as edge computing and smart networks.  The proactive caching in the SCN means placing content in advance, that is popular to a user in an SBS by predicting what content to request.  However, due to the difficulty in predicting the popularity of content on SBSs, the limited cache size, and the content popularity discrepancy among SBSs, the proactive caching in the SCN is the much more difficult problem than caching in a conventional cellular or wired network.
To overcome these limitations and to guarantee users’ Quality of Experience (QoE), this dissertation proposes a new proactive caching framework based on machine learning techniques. First, this dissertation analyzes and summarizes related researches on the proactive caching. Also, the theoretical backgrounds of cooperative communication called Coordinated Multi-Point (CoMP) and various machine learning techniques are described. In the dissertation, the CoMP is mainly used as a transmission technique for efficiently designing the proactive caching and machine learning techniques such as NMF and LSTM are used to form a caching strategy. Next, this dissertation analyzes the caching performance through the CoMP and discusses the fundamental conditions for using the CoMP on the proactive caching in environments where content popularity among SBSs is different. Thirdly, this dissertation conducts a spatial study of the proactive caching based on the machine learning about what contents must be cached in the CoMP-based SCN. Fourth, this dissertation proposes a comprehensive spatio-temporal proactive caching framework using various design principles and machine learning. The performance of the proposed framework is verified through various performance evaluations such as content hit ratio and content delivery completion time.
In conclusion, the wireless resources of the SCN can be saved while satisfying the users’ QoE by knowing, predicting and responding to the user’s needs through the proposed proactive caching framework based on the machine learning. Therefore, network operators and content providers will be able to provide users with high quality services at low cost.