Using the absolute ages of passively evolving galaxies observed at different redshifts, one can obtain the differential ages, the derivative of redshift z with respect to the cosmic time t (i.e. dz/dt). Thus, the Hubble parameter H(z) can be measured through the relation H(z)=−(dz/dt)/(1 + z). By comparing the measured Hubble parameter at different redshifts with the theoretical one containing free cosmological parameters, one can constrain current cosmological models. In this paper, we use this method to present the constraint on a spatially flat Friedmann-Robert-Walker Universe with a matter component and a holographic dark energy component, in which the parameter c plays a significant role in this dark energy model. Firstly we consider three fixed values of c=0.6, 1.0 and 1.4 in the fitting of data. If we set c free, the best fitting values are c=0.26, m0=0.16, h=0.9998. It is shown that the holographic dark energy behaves like a quintom-type at the 1 level. This result is consistent with some other independent cosmological constrains, which imply that c<1.0 is favored. We also test the results derived from the differential ages using another independent method based on the lookback time to galaxy clusters and the age of the universe. It shows that our results are reliable.