Layered LiNi0.4Mn0.4Co0.2O2 has been synthesized by citrate precursor method. Its magnetic properties are investigated by dc magnetization. The high-temperature susceptibility curve follows the Curie-Weiss law with Curie and Weiss constant 1.435(2) emu K/mol Oe and -112(4) K, respectively, larger than those values reported in previous researches, which possibly results from the difference in the synthesis process and sintered temperature. Our dc susceptibility differs from that of the homogeneous spin glass in that below Tirr field cooled (FC) curve continues to rise, while the FC curve is almost flat for homogeneous spin glass. Together with the de Almeida-Thouless line analysis, cluster spin glass is suggested to be the ground state of LiNi0.4Mn0.4Co0.2O2. Frustration parameter |Ø| /Tf in this system is estimated to be about four, lower than the value that frustration effect is strong enough to give rise to spin glass state. This fact indicates that the cluster-spin-glass results from the short-range structure disorder rather than the geometrical frustration.