We propose the second-order ti me discretization scheme with the finite-element approximation for the partial integro-differential equations with a weakly singular kernel. The space discretization is based on the finite element method and the time discretization is based on the Crank-Nicolson scheme with a graded mesh. We show the stability of the scheme and obtain the second-order convergence result for the fully discretized scheme.