We investigate the dephasing of electron wave functions due to interaction with a two-level system (TLS) in a quantum dot. It is found that if the TLS is intimately coupled to the electron rather than being in equilibrium with the thermal bath, the ground state of the whole system is a many-body state with nonzero amplitudes for the TLS being at the higher level. By averaging over the TLS degree of freedom a finite dephasing rate of the pure electron states at zero temperature is obtained. From this we discuss the conditions for observing the saturation of the dephasing rate at low temperatures.