This paper is an article review of "Using the attribute hierarchy method to make diagnostic inferences about examinees’ cognitive skills in algebra on the SAT," which specifically examines the ability to use cognitive 'mapping' to understand the strategies of test-takers when approaching problems on standardized tests. The authors attempt to provide a more useful method of analyzing test responses than simply grading them 'right or wrong' or analyzing test-takers' demographic data.
Attribute Hiearchy
Critique of the Journal Article "Using the Attribute Hierarchy Method to Make Diagnostic Inferences about Examinees' Cognitive Skills in Critical Reading" by Changjiang Wang and Martin J. Gieri
Gierl, M.J., Wang, C., & Zhou, J. (2008). Using the attribute hierarchy method to make diagnostic inferences about examinees' cognitive skills in algebra on the SAT. Journal of Technology, Learning, and Assessment, 6(6). Retrieved from http://www.jtla.org.
One problem with evaluating the effectiveness of different types of test questions is that it is often unclear why students get particular exam questions wrong (or right). The SAT is a particularly controversial and challenging test and can have a long-lasting impact upon a college applicant's life, depending on what score he or she receives. Thus, effective analysis of SAT questions for veracity is essential to be fair to the high school students that take the test.
The purpose of the study by Gierl, Wang & Zhou (2008) entitled "Using the attribute hierarchy method to make diagnostic inferences about examinees' cognitive skills in algebra on the SAT" was an attempt to provide greater clarity about the test-taking strategies used by various students on the SAT than simply analyzing the answers from a correct vs. incorrect perspective. According to the authors, the attribute hierarchy method (AHM) is "a psychometric method for classifying examinees' test item responses into a set of structured attribute patterns associated with different components from a cognitive model of task performance" (Gierl, Wang & Zhou 2008:4). The authors' ultimate objective was to provide a more effective methodology of preparing students as well as evaluating potential test questions, not simply on the SAT but also with various other types of standardized examinations. "Cognitive models provide one method for representing and reporting the examinees' cognitive profile on diverse tasks which could be used to link their weaknesses with instructional methods designed to improve the examinees' skills" (Gierl, Wang & Zhou 2008:6).
When conducting their literature review to substantiate their research, the authors stress that their method of evaluation differs fundamentally from a conventional assessment of student performance or even a summative assessment of the efficacy of an exam. A CDA (cogitative development assessment) relies upon "educational and psychological studies on reasoning, problem solving, and information processing" versus general performance expectations (Gierl, Wang & Zhou 2008:6). The mental approach is literally 'modeled' versus merely determining if questions satisfy the basic expectations of the grade level or ability the exam purports to test.
To examine students' cognitive approaches, the authors deployed a two-step methodology in their procedures. First, SAT algebra questions were selected specifically for targeted analysis. Then, a "cognitive model was developed by having content specialists…review the SAT algebra items, identify their salient attributes, and order the item-based attributes into a hierarchy. Then, the cognitive model was validated by having a sample of students think aloud as they solved each item" (Gierl, Wang & Zhou 2008:4). Secondly, a quantitative, psychometric analysis was used to see if the data fit the expected response patterns of this cognitive model, based upon "observed response patterns produced from a random sample of 5000 examinees who wrote the items. Attribute probabilities were also computed for this random sample of examinees so diagnostic inferences about their attribute-level performances could be made" (Gierl, Wang & Zhou 2008:4).
The model used was constructed via a four-step process. First, the model was formed, "using psychometric methods and linking these skills to diagnostic inferences" (Gierl, Wang & Zhou 2008: 39). Secondly, four models were elaborated "that describe different aspects of problem solving using sample items from Algebra I and II" and one was specifically selected for the study (Gierl, Wang & Zhou 2008: 39). "The third aspect, model use, provides structure to the model so that explanations and predictions can be made" and then fourth step was evaluation of the model (Gierl, Wang & Zhou 2008: 39).
The scope of the study was admittedly rather narrow -- only one model was tested. The researchers admitted that more testing was needed in this area of research and the study was a relatively preliminary experiment in mind-mapping, Student skills needed to be broken down into more specific attributes and a further problem was that the study was retroactive, which they hoped would not be the case in the future: "this order of events -- where the cognitive model is first identified and then the test items are developed -- is needed because the hierarchical organization of attributes should guide the development of test items and, subsequently, the interpretation of test performance when using the AHM" (Gierl, Wang & Zhou 2008: 44).
You’re 84% through this paper. Sign up to read the full paper.
Sign Up Now — Instant Access Already a member? Log inAlways verify citation format against your institution’s current style guide requirements.