A full quantitative evaluation of the depolarization of light may serve to assess concentrations of depolarizing particles in the retinal pigment epithelium and to investigate their role in retinal diseases in the human eye. Optical coherence tomography and optical frequency domain imaging use spatial incoherent averaging to compute depolarization. Depolarization depends on accurate measurements of the polarization states at the receiver but also on the polarization state incident upon and within the tissue. Neglecting this dependence can result in artifacts and renders depolarization measurements vulnerable to birefringence in the system and in the sample. In this work, we discuss the challenges associated with using a single input polarization state and traditional depolarization metrics such as the degree‐of‐polarization and depolarization power. We demonstrate quantitative depolarization measurements based on Jones vector synthesis and polar decomposition using fiber‐based polarization‐sensitive optical frequency domain imaging of the retinal pigment epithelium in a human eye.