In this dissertation, we address formal analysis of efficient multicast in IP networks. The multicast is a effective mechanism for communicating among large number of group members. First, we present a BestRelay relation for constructing multicast overlay tree. It can be used for a reliable multicast retransmission tree over IP multicast network and also for a overlay multicast relaying tree over IP unicast routing. Efficiency of retransmission trees for a reliable multicast is mainly depends on locations of repairers, which are in charge of retransmitting lost packets. We propose an algorithm for each receiver to find a repairer for efficient recovery. The resulting tree for retransmission is organized by pairs of a receiver and a repairer which is the host "nearest" to the receiver among the multicast group members "nearer" to the sender. We formally prove that the proposed algorithm realizes reliable multicast with only constant times of a lower bound cost achievable through impractical router support. We also evaluate the algorithm through extensive simulations. Overlay multicast has been proposed for implementing multicast in the application layer as a practical alternative of IP multicast. Since a packet in overlay multicast is transmitted via unicast between end hosts, the same packet can be sent over a link multiple times, and thus, the main challenge of overlay multicast is how to organize multicast delivery trees to minimize bandwidth consumption. In this dissertation, we present a formal analysis of bandwidth efficiency of overlay multicast using a simple tree construction algorithm using BestRelay. Our analytic result shows that the bandwidth consumption of overlay multicast can be bounded by twice of that of optimal IP multicast. We confirm this result through simulation as well.