A relay-based multiple-antenna wireless communication system can increase system capacity and cell coverage. In addition, that makes it possible to support transmission for multiple users located far from the base station. It is necessary to precode the signal vector at both base station and relay station to support multi-user transmission in downlink cellular systems and those precoding schemes are very similar to conventional precoding schemes in multi-user MIMO (MU-MIMO) downlink. In this thesis, we propose the two novel precoding schemes at base station and relay station. The first one is linear optimal precoders which minimize mean square error (MSE) of received signal. The optimal precoders are obtained by solving an unconstraint optimization problem. This problems are coupled to each other, therefore, closed-form solutions do not exist. Thus, each solution is obtained by the proposed iterative algorithm. The second one is to apply one of the MU-MIMO precoding schemes using the lattice reduction (LR) algorithm and modulo operation to our amplify and forward (AF) MIMO relaying system. We perform the average BER performance comparison between the proposed linear optimal scheme and the conventional precoding schemes such as zero forcing (ZF)-precoding and Wiener precoding. And we also check that the proposed LR-aided precoding schemes have the achievable rate which is superior to that of the conventional ZF-precoding.