Accurate prediction of the visual attributes is significant in various recognition tasks. For many visual attributes, while it is very difficult to describe the exact degrees of their presences, by comparing the pairs of samples, the relative ordering of presences may be easily figured out. Based on this observation, instead of considering such attribute as binary attribute, the relative attribute method learns a ranking function for each attribute to provide more accurate and informative prediction results. In this paper, we also explore pairwise ranking for visual attribute prediction and propose to improve the relative attribute method in two aspects. First, we propose a relative tree method, which can achieve more accurate ranking in case of nonlinearly distributed visual data. Second, by resorting to randomization and ensemble learning, the relative tree method is extended to the relative forest method to further boost the accuracy and simultaneously reduce the computational cost. To validate the effectiveness of the proposed methods, we conduct extensive experiments on four databases: PubFig, OSR, FGNET, and WebFace. The results show that the proposed relative forest method not only outperforms the original relative attribute method, but also achieve the state-of-the-art accuracy for ordinal visual attribute prediction.