The present disclosure relates to a pre-5th-Generation (5G) or 5G communication system to be provided for supporting higher data rates Beyond 4th-Generation (4G) communication system such as Long Term Evolution (LTE). According to various embodiments in the present disclosure, an operating method of a base station in a wireless communication system may include determining a transmission mode of the greatest channel capacity among a plurality of transmission modes based on first channel information of a first terminal and second channel information of a second terminal, and transmitting a transmit signal generated based on the determined transmission mode to the first terminal and the second terminal. The first transmission mode of the plurality of the transmission modes may be a transmission mode for applying a diversity scheme to a first signal for the first terminal, applying a multiplexing scheme to a second signal for the second terminal, and transmitting the transmit signal comprising the first signal and the second signal in a non-orthogonal multiple access (NOMA) scheme.