Heterogeneous Network (HetNet) has been a major feature for next generation wireless networks, where femtocell has attracted much attention as the most promising radio access technology (RAT) for providing improvements on network capacity,
service coverage and power efficiency. To overcome the scarcity of licensed spectrum, utilization of unlicensed spectrum bands where wireless fidelity (Wi-Fi) operated triggered the design for coexistence of Long Term Evolution (LTE) and Wi-Fi in LTE
unlicensed band (LTE-U). It is shown that analysis by Voronoi diagram achieves superior performance under homogeneous environment. In this paper, we extend the Voronoi diagram to HetNet environment with Wi-Fi and femtocells. For the HetNet, we formulate an optimization problem, which is obtained from a weighted Voronoi tessellation and the corresponding characteristic functions, to maximize network capacity over an optimal femtocell operation scheme in LTE-U. For the optimal femtocell operation, we provide an optimal AP density ratio under the objective of HetNet capacity maximization with convex analysis, where partial femto APs are turned to open access mode in the HetNet coverage. Simulation results are provided to demonstrate effectiveness of the proposed algorithm, and validate importance of the AP density ratio and the transmit power ratio to network capacity. This analysis