The dynamical behaviors of a nonlinear Brownian oscillator are studied using Fokker-Planck equation. From the Fokker-Planck equation, the equation for the displacement autocorrelation function and that for the spectral density function are derived by use of the projection operator method and the matrix continued-fraction method. From the numerically evaluated spectral density the vibrational dephasing time is obtained, and its dependence on the friction-constant and the anharmonic term in the potential function is investigated. Furthermore we show that the numerical results from the projection operator method and the matrix continued-fraction method agree with each other up to the friction constant $\alpha=0.5$.