The understanding of the mineralization of collagen for bone formation is a current key theme in bone tissue engineering and is of great relevance to the fabrication of novel biomimetic bone grafting materials. The noncollagenous proteins (NCPs) play a vital role in bone formation and are considered to be responsible for regulating intrafibrillar penetration of minerals into collagen fibrils by means of their abundant polyanionic domains. In this study, alginate, as a NCPs analogue, was introduced in the mineralization of collagen to mediate the collagen self-assembly with simultaneous hydroxyapatite (HA) synthesis. The biomimetic systems were based upon the self-assembly of collagen (Col) or collagen-alginate (CA) in the absence or presence of a varying content of HA. The alginate-mediated effects were found to include the lateral aggregation of small fibrils into the extremely large bundles and the assisted deposition of HA for a larger mineralized fibril. This alginate-assisted mineralization of collagen gave rise to an exquisite 3D mineralized architecture with enhanced mechanical property. The cell viability experiments showed the excellent proliferation and spreading morphologies of rat bone mesenchymal stem cells (MSCs) on the assembled products, and a higher expression of osteogenic differentiation related transcription factor was obtained in the alginate-assisted mineralization of collagen. This study indicated that the selection of an appropriate substance, e.g., alginate as an anionic polyelectrolyte with Ca-capturing property, could be a convenient, simple solution to achieve a mineralized collagen scaffold with the reinforced mechanical property for potential applications in bone regeneration.