Structural control of solubilizing side groups in fullerene-based electron acceptors is critically important to optimize their performance in bulk heterojunction (BHJ)-type polymer solar cell (PSC) devices. The structural changes of fullerene derivatives affect not only their optical and electrochemical properties but also their solubility and miscibility with electron donor polymers. Herein, we synthesized a series of o-xylenyl C-60 bis-adduct (OXCBA) derivatives with different solubilizing side groups to systematically investigate the effects of fullerene derivative structures on the photovoltaic properties of PSCs. The xylenyl side groups on the OXCBA were modified to produce several different OXCBA derivatives in which the xylenyl groups were functionalized with fluorine (FXCBA), nitro (NXCBA), methoxy and bromine (BMXCBA), and phenyl groups (ACBA). End group modifications of OXCBA dramatically affect photovoltaic performance in blend films with poly(3-hexylthiophene) (P3HT), resulting in power conversion efficiencies (PCEs) ranging from 1.7 to 5.3%. We found that this large range in PCE values is mainly due to differences in the blend morphology and interfacial area of the P3HT:OXCBA derivative films caused by changes in the hydrophobicity of the OXCBA derivatives and their interaction with P3HT. The trend in photovoltaic performance of the different OXCBA derivatives agrees well with those of the interfacial tension, PL quenching, and exciton dissociation probability, which suggests that changes in the interaction with P3HT are largely responsible for their photovoltaic performances. Finally, the OXCBA derivatives were applied in inverted type PSC devices. We note that P3HT:OXCBA blend devices exhibited more than 5% PCE with excellent air stability, which is one of the best inverted type devices based on the P3HT polymer in a simple device architecture without any extra interlayers.