The conductivity of Fe3O4 drops two orders of magnitude below the Verwey temperature Tv, known as the Verwey transition, due to the formation of charge ordering (CO). Here, we report the discovery of a large birefringence effect correlated with the CO in Fe3O4 controlled by ultrafast-laser-assisted magnetic field cooling. The polarization rotation (PR) of the light reflected from a single crystalline Fe3O4 film below Tv shows a twofold symmetry as the cooling field (CF) rotates through 360∘ within the film plane. The maximum PR occurs for the CF parallel to the cubic ⟨110⟩ axes, and its amplitude depends on the sample orientation. These results are well interpreted by taking into account two CO patterns with orthogonal CO orientations, with their fractional areas determined by the ratio of the field components along the [110] and [1¯10] axes. Our results indicate that application of the CF along ⟨110⟩ axes may result in the single orientation CO state, which is highly desirable for unraveling the subtle CO structure to better understand the driving mechanism of the Verwey transition. In addition, ultrafast pump-probe measurements reveal a diminishment of the twofold PR at 0.8 ps due to fast melting of the CO state by the ultrafast laser pulse.