This paper proposes a Markov decision process (MDP) based base station management scheme that dynamically and collectively manages activation of a group of base stations depending on the time-varying traffic demand for power conservation in self-organization networks. Our MDP model is unique in a sense that it accurately captures the dynamics of handover traffic among neighboring cells, and it formulates infeasible actions as constraints in a constrained optimization problem. Simulation results confirm that the proposed scheme can significantly reduce power consumption: 55% of daily power savings, upto 73% of power savings during low traffic periods, and the minimum 23% of power savings even during high traffic periods. Our MDP algorithms find desired optimal policies to deactivate unnecessary base stations without sacrificing network performances.