Packet voice conference system allows participants to talk freely with one another in packet switched networks. To obtain this free talking, some proper algorithm is needed to manage arriving voice packets from senders. The proposed algorithm in this work estimates the target playback times of voice packets and determines the set of voice packets in the same set mixed into one voice data, and sent to a D/A converter. The proposed algorithm can manage the multiple voice streams which could occur simultaneously in packet voice conference, and does not lose the intra-synchronization of each voice stream. Because voice connections in packet voice conference is established through various networks, the property of each sender``s voice packets are different. This may result in interference among voice connections. The proposed algorithm minimizes the effect of interference, and establishes independent voice connections. The proposed algorithm analyzes the voice streams which arrive at the receiver, and minimizes the voice delay of each voice connection by the proper feedback as the change of network situation. Finally, this paper evaluates the performance of the proposed algorithm by mathematical analysis, and makes sure of correct analysis by some simulations.