As millimeter wave (mmWave) beamforming (BF) system can increase the areal capacity in ultra-dense network (UDN), a joint design of user association and hybrid BF is challenging due to the high feedback overhead and complicated interference management where many communication nodes use large antenna arrays. In addition, the wireless backhaul should be incorporated because it allows a flexible design, while the wired backhaul is difficult to be installed as the number of small base stations increases. To integrate the wireless backhaul and access, the time resource partitioning between the backhaul and access transmissions is needed, which affects the network performance combined with the user association and BF design. In this regard, this paper deals with the joint problem of the time resource allocation, user association, and BF design to maximize the weighted sum rate. Then a two-stage design approach is proposed based on the limited channel feedback to reduce the complexity and overhead. Simulation results show that the proposed limited feedback hybrid BF scheme outperforms the baseline schemes. In addition, we find that the maximum spatial multiplexing gain can be achieved if the number of antennas of each node increases with an order of larger than the square of the number of RF chains. At the same time, the multi-node diversity gain can be obtained in mmWave UDN, which depends on the path loss exponent of line-of-sight.