This paper is an article review of the quantitative educational article "Integrating test-form formatting into automated test assembly." The authors examine a new technique used to both generate problems for tests and also to create the test forms in a simultaneous fashion. This critique first addresses the substance and then the style of the article.
Automated Test Form Generation
Critique of the Journal Article "Automated Test Form Generation" by Wim J. van der Linden and Qi Diao
Van der Linden, J.M. & Diao, Q. (2013). Integrating test-form formatting into automated test assembly. Applied Psychological Measurement, 37(5) 361 -- 374
Retrieved: http://www.utwente.nl/gw/omd/Medewerkers/artikelen/vdLinden/artikelen%20vanaf%20juni%202011/APM%202013,%20361-374.pdf
The ordering of items in test forms and the selection of test items from a database can be extremely tedious and laborious when done manually. It would be extremely advantageous to be able to do both simultaneously using automated test form generation (ATFG). But the ATFG process can be unwieldy and cumbersome given the number of variables the creator must tackle. For example, "for an item bank of size 2,000, test length of 50 items, and upper limit of the number of pages in the test form equal to 15, the number of variables in the model is equal to 1,500,000" (Van der Linden & Diao 2013: 361). According to the article, "Integrating test-form formatting into automate test assembly" from Applied Psychological Measurement, "one of the goals of this research is to find out if test-production problems of realistic sizes can be solved" and make the simultaneous generation process easier for test designers (Van der Linden & Diao 2013: 361).
The process of automated test form generation consists of two steps. The first is the selection of items from a database and ordering and formatting the item into the actual test form itself. "ATA involves the selection of the items for a test form from an item bank subject to a potentially large set of content and statistical specifications while optimizing their measurement properties" through the use of mixed integer programming (MIP)" (Van der Linden & Diao 2013: 361). While MIP is mainly used for test selection, it can also "be used to optimize the second step (automated test-form generation [ATFG]" (Van der Linden & Diao, 2013: 361). The purpose of the study is to determine how this can be accomplished in an efficient manner. The objective is to explore whether it is possible to use ATA and ATFG simultaneously for more efficient test generation and "then evaluate their performances against the case of separate models for ATA and ATFG for a series of applications with realistic item bank sizes and test lengths" (Van der Linden & Diao, 2013: 361).
Previous authors have found this quite challenging so rather than a formal literature review, the authors construct two paradigmatic examples of the problems they are trying to solve in their article and demonstrate that their simultaneous approach is both possible and more efficient than a two-step process. They address the problems of generating tests consisting of problems in computer-based and paper-based formats. With computers, "in real-world applications, multiple versions of these constraints are needed to deal with multiple categorical and quantitative item attributes, as well as possibly different additional types of constraints, for instance, to deal with items that share a stimulus" (Van der Linden & Diao, 2013: 364). Rather than less complicated, paper formats are even more difficult to assemble in many ways. "Paper forms tend to be more complicated to assemble…because we need to observe page breaks and typically want to minimize the use of paper" as well as an adequate array of problems for different versions of tests that still draw upon all the required materials (Van der Linden & Diao, 2013: 364).
To argue that to optimize ATFG is possible, the authors simulated two different test forms: one computerized and the other paper based, according to various "content, statistical, and formatting specifications. More specifically, they used an existing set of specifications for the assembly of test forms, along with an existing item pool from a large-scale mathematics achievement testing program" (Van der Linden & Diao, 2013: 368). This was to show that ATFG was both hypothetically possible as well as applicable to the needs of an actual testing program
In their conclusion, the authors noted that it was possible to deploy a simultaneous modeling approach that allowed an integration of both selection of problems and generation of test forms, considerably streamlining existing available formats. "For future applications, the performance of the solver needs to be assessed on a case-by-case basis. The total number of decision variables is only one of the factors with an impact on the performance of MIP solvers; others are the structure and sparseness of the coefficient matrix for the constraints" (Van der Linden & Diao, 2013: 368). Thus, the simultaneous approach is recommended, based upon their calculations although this may vary according to circumstances on occasion.
You’re 85% through this paper. Sign up to read the full paper.
Sign Up Now — Instant Access Already a member? Log inAlways verify citation format against your institution’s current style guide requirements.