A nonlinear dispersion relation is presented to model the nonlinear dispersion of waves over the whole range of possible water depths. It reduces the phase speed over-prediction of both Hedges' modified relation and Kirby and Dalrymple's modified relation in the region of 1 < kh <1.5 at small wave steepness and maintains the monotonicity in phase speed variation at large wave teepness. And it has a simple form. By use of the new nonlinear dispersion relation along with the mild slope equation taking into account weak nonlinearity, a mathematical model of wave transformation is developed and applied to laboratory data. The results show that the model with the new dispersion relation can predict the wave transformation over complicated bathymetry satisfactorily.