Current all polymer solar cells still suffer from low fill factors (FF) and short-circuit current density (Jsc) compared with the conventional polymer/fullerene system. Herein in this work, devices using PTP8 as the electron donor and [70]PCBM as well as widely used polymer N2200 as the electron acceptor were systematically studied and compared. The major loss mechanisms in the all polymer solar cells were investigated to understand their relatively lower performance than the PTP8/fullerene system. By performing in-depth analysis on ultrafast transient transmission spectroscopy results, we estimated that in PTP8/N2200 device nearly half of the charges recombine geminately, which is confirmed as the major factor hindering the device performance of all polymer solar cells compared with polymer/fullerene system. Through thorough morphology analysis, the low charge generation efficiency is attributed to the reduced crystallinity of N2200 in the blend film and the unfavorable face-to-edge orientation at the donor/acceptor heterojunction. Coupling these results with knowledge from efficient polymer/fullerene systems, the future design of new polymers can devote to increase the attraction between the π face of donor and acceptor, leading to enhanced face-to-face orientation at the heterojunction, while maintaining a high π-π stacking order for each polymer.