This paper presents a novel computer-aided diagnosis (CAD) system for the diagnosis of breast cancer based on extreme learning machine (ELM). In view of a mammographic image, it is first eliminated interference in the preprocessing stages. Then, the preprocessed images are segmented by the level set model we proposed. Subsequently, a model of multidimensional feature vectors is built. Since not every feature vector contributes to the improvement of performance, feature selection is done by the combination of support vector machine (SVM) and extreme learning machine (ELM). Finally, an optimal subset of feature vectors is inputted into the classifiers for distinguishing malignant masses from benign ones. We also compare our breast mass classification approach based on ELM with several state-of-the-art classification models, and the results show that the proposed CAD system not only has good performance in terms of specificity, sensitivity and accuracy, but also achieves a significant reduction in training time compared with SVM and particle swarm optimization-support vector machine (PSO-SVM). Ultimately, our system achieves the better performance with average accuracy of 96.02% which indicates that the proposed segmentation model, the utilization of selected feature vectors and the effective classifier ELM provide satisfactory system.