Ultrafast transient absorption (TA) spectroscopy was employed to investigate the thermal annealing effect on the charge transfer (CT) in bulk heterojunction (BHJ) all-polymer solar cells (all-PSCs) utilizing an n-type polymer P(NDI2OD-T2) (Polyera, N2200) as acceptor and a low bandgap polymer PBPT as donor. The CT generates hole polarons residing in the PBPT and electron polarons belonging to N2200, manifested in the TA spectra of the BHJ films as the long-lived absorption peak centered at ∼850 nm. The CT is most efficient in the film annealed at 160 °C and its efficiency declines monotonically when enhancing or reducing the annealing temperature, displaying a positive correlation with the power conversion efficiency (PCE) of the corresponding solar cell devices. This correlation is analyzed in terms of the crystallinity and phase separation, which are the key factors determining the performance of all-PSCs. Our results can provide valuable guidance for the fabrication of BHJ all-PSCs to improve their PCE.