Attribute Hierarchy Method for SAT Diagnostic Inferences
This paper critiques Gierl, Wang, and Zhou's (2008) study, which applied the attribute hierarchy method (AHM) to diagnose examinees' cognitive skills in SAT algebra. The critique examines the study's purpose, methodology, cognitive development assessment framework, and limitations. The AHM classifies student responses into structured attribute patterns derived from a cognitive model of task performance, offering a more nuanced alternative to simple correct-versus-incorrect scoring. The paper evaluates the study's four-step model construction process, its use of think-aloud protocols and psychometric analysis with 5,000 examinees, and ultimately questions whether the study's narrow scope, retroactive design, and limited generalizability undermine its ambitious research goals.
- Introduction: SAT's impact and need for fair question analysis
- Purpose and Background of the Study: AHM defined and study objectives explained
- Literature Review and Cognitive Development Assessment: CDA contrasted with conventional performance assessment
- Research Methodology and Model Construction: Two-step and four-step model procedures described
- Scope and Limitations: Narrow scope, retroactive design, and attribute issues
- Overall Critique and Conclusion: Ambition exceeds findings; generalizability questioned
✍️ How to write this paper — guide, tools & examples ▾
What makes this paper effective
- The critique follows the structure of the source article closely, moving logically from purpose and background through methodology to evaluation, making the argument easy to track.
- Frequent direct quotation from the source article grounds each analytical point in specific textual evidence rather than paraphrase alone.
- The concluding critique is appropriately measured — acknowledging the study's intriguing elements while clearly identifying weaknesses such as narrow scope, retroactive design, and limited generalizability.
Key academic technique demonstrated
The paper demonstrates disciplined source integration: it uses block quotations with precise page references to substantiate descriptive claims, then transitions to evaluative commentary. This technique — summarize, cite, evaluate — is the foundational move of an academic article critique and is executed consistently throughout.
Structure breakdown
The paper opens by establishing the significance of SAT testing, then introduces the AHM framework and its purpose. It moves through the source study's literature review rationale, two-step methodology, and four-step model construction before narrowing to admitted limitations. The final paragraph delivers an evaluative judgment covering scope, generalizability, and design feasibility, giving the critique a clear arc from description to assessment.
Introduction
One problem with evaluating the effectiveness of different types of test questions is that it is often unclear why students get particular exam questions wrong — or right. The SAT is a particularly controversial and challenging test that can have a long-lasting impact on a college applicant's life, depending on the score received. Thus, effective analysis of SAT questions for validity is essential to ensure fairness for the high school students who take the test.
Purpose and Background of the Study
The purpose of the study by Gierl, Wang, and Zhou (2008), entitled "Using the Attribute Hierarchy Method to Make Diagnostic Inferences about Examinees' Cognitive Skills in Algebra on the SAT," was to provide greater clarity about the test-taking strategies used by various students on the SAT, going beyond a simple correct-versus-incorrect perspective. According to the authors, the attribute hierarchy method (AHM) is "a psychometric method for classifying examinees' test item responses into a set of structured attribute patterns associated with different components from a cognitive model of task performance" (Gierl, Wang & Zhou, 2008, p. 4).
The authors' ultimate objective was to provide a more effective methodology for preparing students as well as evaluating potential test questions — not simply on the SAT, but across various other types of standardized examinations. As they explain, "cognitive models provide one method for representing and reporting the examinees' cognitive profile on diverse tasks, which could be used to link their weaknesses with instructional methods designed to improve the examinees' skills" (Gierl, Wang & Zhou, 2008, p. 6).
Literature Review and Cognitive Development Assessment
When conducting their literature review to substantiate their research, the authors stress that their method of evaluation differs fundamentally from a conventional assessment of student performance or even a summative assessment of exam efficacy. A cognitive development assessment (CDA) relies upon "educational and psychological studies on reasoning, problem solving, and information processing" rather than general performance expectations (Gierl, Wang & Zhou, 2008, p. 6). The mental approach is literally "modeled" rather than merely determining whether questions satisfy the basic expectations of the grade level or ability the exam purports to test.
Research Methodology and Model Construction
To examine students' cognitive approaches, the authors deployed a two-step methodology. First, SAT algebra questions were selected specifically for targeted analysis. Then, a "cognitive model was developed by having content specialists review the SAT algebra items, identify their salient attributes, and order the item-based attributes into a hierarchy. The cognitive model was then validated by having a sample of students think aloud as they solved each item" (Gierl, Wang & Zhou, 2008, p. 4).
Secondly, a quantitative, psychometric analysis was used to assess whether the data fit the expected response patterns of this cognitive model, based upon "observed response patterns produced from a random sample of 5,000 examinees who completed the items. Attribute probabilities were also computed for this random sample of examinees so that diagnostic inferences about their attribute-level performances could be made" (Gierl, Wang & Zhou, 2008, p. 4).
The model itself was constructed via a four-step process. First, the model was formed "using psychometric methods and linking these skills to diagnostic inferences" (Gierl, Wang & Zhou, 2008, p. 39). Second, four models were elaborated "that describe different aspects of problem solving using sample items from Algebra I and II," and one was specifically selected for the study (Gierl, Wang & Zhou, 2008, p. 39). "The third aspect, model use, provides structure to the model so that explanations and predictions can be made," and the fourth step was the evaluation of the model (Gierl, Wang & Zhou, 2008, p. 39).
Create your account
Always verify citation format against your institution’s current style guide requirements.