An Evaluation of Cognitive Diagnostic Assessment Attribute-Level Classification Consistency and Accuracy Indices

• 1、College of Computer Information and Engineering, Jiangxi Normal University, Nanchang, 330022
• 2、Elementary Educational College, Jiangxi Normal University, Nanchang, 330027
• 3、State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing 100875

Abstract：Classification consistency and accuracy indices can be used as important indicators of the reliability and validity of classi?cation results. Cognitive diagnostic assessment focuses on make attribute classification-based decisions while reliability of classification is often not reported in diagnostic score reporting. Classification consistency and accuracy indices (Cui, et al., 2012) were used to evaluate the consistency and accuracy of classification results at the whole-pattern level, but it needs to develop attribute-level classification consistency and accuracy indices for CDA(Cui, et al., 2012).The study investigated attribute-level and pattern-level classification consistency and accuracy indices based on the Rudner method or the Guo method in item response theory. Procedures were developed for the computation of classification consistency and accuracy indices specifically designed for cognitive diagnostic assessments. We also used simulated data from the deterministic input noisy "and" gate model to evaluate their performances. According to the simulation results, the classification accuracy indices performed well with simulated diagnostic tests in that their values matched closely with the simulated correct classification rates across different simulation conditions. And classification consistency indices matched with the results of the test-retest consistency indices .

Keywords： educational psychology classification consistency index classification accuracy index empirical reliability of attribute；Cronbach's α reliability of attribute；the deterministic input noisy