We point out using an empirical tight-binding approach that the ground state of holes in InAs/GaAs self-assembled quantum dots carries nonzero orbital momentum. The spin and orbital motions of the hole state are found to have opposite contributions to the hole g factor, leading to zero g factors of holes and then excitons in dots of high aspect ratio. The nonzero envelope orbital momenta of the holes are also shown to account for anisotropic circular polarization of exciton emission and nonlinear Zeeman splittings in high magnetic fields. Our theory well explains recent experiments and indicates the possibility of engineering magnetic splitting by tuning the electric confinement in nanostructures.