Testing (over) dominance as the genetic cause of heterosis and estimating the (over)dominance coefficient (h) are related. Using simulations, we investigate the statistical properties of Mukai's approach, which is intended to estimate the average (h) of hi across loci by regression of outcrossed progeny on the sum of the two corresponding homozygous parents. A new approach for estimating h is also developed, utilizing data on families formed by multiple selfed genotypes from each outcrossed parent, thus not requiring constructing homozygotes. Assuming constant mutation effects, h can be estimated accurately by both approaches under dominance. When rare alleles have low frequencies at any polymorphic locus, Mukai's approach can estimate h accurately under over (under) dominance. Therefore, the (over) dominance hypothesis for heterosis can be tested by estimating h, under either dominance or overdominance at all genomic loci. However, this is invalid with more plausible mixed dominance and overdominance at different loci. Estimating the variance of hi across loci is also investigated. In self-compatible outcrossing populations with mutations of variable effects and lethals, our new approach is better than Mukai's, not only because of not requiring homozygotes but also because of the better statistical performance reflected by the smaller mean square errors of the estimates.