Although epistasis is an important phenomenon in the genetics and evolution of complex traits, epistatic effects are hard to estimate. The main problem is due to the overparameterized epistatic genetic models. An epistatic genetic model should include potential pair-wise interaction effects of all loci. However, the model is saturated quickly as the number of loci increases. Therefore, a variable selection technique is usually considered to exclude those interactions with negligible effects. With such techniques, we may run a high risk of missing some important interaction effects by not fully exploring the extremely large parameter space of models. We develop a penalized maximum likelihood method. The method developed here adopts a penalty that depends on the values of the parameters. The penalized likelihood method allows spurious QTL effects to be shrunk towards zero, while QTL with large effects are estimated with virtually no shrinkage. A simulation study shows that the new method can handle a model with a number of effects 15 times larger than the sample size. Simulation studies also show that results of the penalized likelihood method are comparable to the Bayesian shrinkage analysis, but the computational speed of the penalized method is orders of magnitude faster.

In plants and laboratory animals, QTL mapping is commonly performed using F2 or BC individuals derived from the cross of two inbred lines. Typical QTL mapping statistics assume that each F2 individual is genotyped for the markers and phenotyped for the trait. For plant traits with low heritability, it has been suggested to use the average phenotypic values of F3 progeny derived from selfing F2 plants in place of the F2 phenotype itself. All F3 progeny derived from the same F2 plant belong to the same F2: 3 family, denoted by F2: 3. If the size of each F2: 3 family (the number of F3 progeny) is sufficiently large, the average value of the family will represent the genotypic value of the F2 plant, and thus the power of QTL mapping may be significantly increased. The strategy of using F2 marker genotypes and F3 average phenotypes for QTL mapping in plants is quite similar to the daughter design of QTL mapping in dairy cattle. We study the fundamental principle of the plant version of the daughter design and develop a new statistical method to map QTL under this F2: 3 strategy. We also propose to combine both the F2 phenotypes and the F2: 3 average phenotypes to further increase the power of QTL mapping. The statistical method developed in this study differs from published ones in that the new method fully takes advantage of the mixture distribution for F2: 3 families of heterozygous F2 plants. Incorporation of this new information has significantly increased the statistical power of QTL detection relative to the classical F2 design, even if only a single F3 progeny is collected from each F2: 3 family. The mixture model is developed on the basis of a single-QTL model and implemented via the EM algorithm. Substantial computer simulation was conducted to demonstrate the improved efficiency of the mixture model. Extension of the mixture model to multiple QTL analysis is developed using a Bayesian approach. The computer program performing the Bayesian analysis of the simulated data is available to users for real data analysis.

Many commercial inbred lines are available in crops. A large amount of genetic variation is preserved among these lines. The genealogical history of the inbred lines is usually well documented. However, quantitative trait loci (QTL) responsible for the genetic variances among the lines are largely unexplored due to lack of statistical methods. In this study, we show that the pedigree information of the lines along with the trait values and marker information can be used to map QTL without the need of further crossing experiments. We develop a Monte Carlo method to estimate locus-specific identity-by-descent (IBD) matrices. These IBD matrices are further incorporated into a mixed-model equation for variance component analysis. QTL variance is estimated and tested at every putative position of the genome. The actual QTL are detected by scanning the entire genome. Applying this new method to a well-documented pedigree of maize (Zea mays L.) that consists of 404 inbred lines, we mapped eight QTL for the maize male flowering trait, growing degree day heat units to pollen shedding (GDUSHD). These detected QTL contributed >80% of the variance observed among the inbred lines. The QTL were then used to evaluate all the inbred lines using the best linear unbiased prediction (BLUP) technique. Superior lines were selected according to the estimated QTL allelic values, a technique called marker-assisted selection (MAS). The MAS procedure implemented via BLUP may be routinely used by breeders to select superior lines and line combinations for development of new cultivars.

Mapping multiple QTL is a typical problem of variable selection in an oversaturated model because the potential number of QTL can be substantially larger than the sample size. Currently, model selection is still the most effective approach to mapping multiple QTL, although further research is needed. An alternative approach to analyzing an oversaturated model is the shrinkage estimation in which all candidate variables are included in the model but their estimated effects are forced to shrink toward zero. In contrast to the usual shrinkage estimation where all model effects are shrunk by the same factor, we develop a Bayesian method that allows the shrinkage factor to vary across different effects. The new shrinkage method forces marker intervals that contain no QTL to have estimated effects close to zero whereas intervals containing notable QTL have estimated effects subject to virtually no shrinkage. We demonstrate the method using both simulated and real data for QTL mapping. A simulation experiment with 500 backcross (BC) individuals showed that the method can localize closely linked QTL and QTL with effects as small as 1% of the phenotypic variance of the trait. The method was also used to map QTL responsible for wound healing in a family of a (MRL/MPJ

Many disease resistance traits in plants have a polygenic background and the disease phenotypes are modified by environmental factors. As a consequence, the phenotypic values usually show a quantitative variation. The phenotypes of such disease traits, however, are often measured in discrete but ordered categories. These traits are called ordinal traits. In terms of disease resistance, they are called quantitative resistance traits, as opposed to qualitative resistance traits, and are controlled by the quantitative resistance loci (QRL). Classical quantitative trait locus mapping methods are not optimal for ordinal trait analysis because the assumption of normal distribution is violated. Methods for mapping binary trait loci are not suitable either because there are more than two categories in ordinal traits. We developed a maximum likelihood method to map these QRL. The method is implemented via a multicycle expectationconditional-maximization (ECM) algorithm under the threshold model, where we can estimate both the QRL effects and the thresholds that link the disease liability and the categorical phenotype. The method is verified in simulated data under various combinations of the parameters. An SAS program is available to implement the multicycle ECM algorithm.