To investigate the relations between the characteristics of biological fouling layer and the fouling rate of membrane in membrane bioreactors used for wastewater treatment, a comprehensive study by atomic force microscopy (AFM), scanning electron microscopy (SEM) and high pressure liquid chromatography (HPLC) was carried out with comparative analysis of two fouled membranes which operated in mixed activated sludge liquor and its supernatant, respectively. The layer surface morphology was characterized with SEM and AFM. Further, to discuss the layer formation mechanism, the molecular weight distribution of soluble microbial products (SMPs) was investigated with HPLC. The results show that AFM is an appropriate technique to characterize the biological fouling layer morphology at nanometer level. AFM reveals the layer microstructure characteristics greatly relate to the fouling rate of membrane by three-dimensional images, profile analyses and roughness computations. In the mixed activated sludge liquor, the membrane with the better filterability has rougher fouling layer. The rougher fouling layer is contributed to the role of bigger microbial flocs, which can prevent the SMPs from directly attaching on the membrane surface and keep holes in the flocs overlaps.