Evaluating the SAT's predictive validity and bias claims

¶ … SAT

GENERAL INFORMATION

Under the helping hands of school counselors, parents, and teachers, more than 1.2 million high school seniors took the Educational Testing Service's SAT in 2008 (Zemelman et al., 2008). ETS makes and administers the aptitude test for the College Board, a nationwide consortium of colleges and universities. But for all practical purposes, this is a monolithic SAT organization. Call it the ETS/College Board alliance. Besides the SAT, an additional one million seniors took its Iowa City cousin, the ACT college admissions test, produced by ACT Inc. In either case, students, parents, and counselors have been subjected to the relentless message that the tests (Blumer, 2004) although not infallible, of course; are good predictors of one's prospects for college success. That has been the take-home message of the admissions testing industry's sales force for decades, and little has changed in recent years. For example, a sampling of ETS and College Board statements in 2008 about the relative merits of its SAT I (the verbal and math reasoning test) includes the following:

The ETS: "High school grades have great value, but they are subject to variability from place to place. Standardized admissions tests offer colleges and universities a fair and impartial way to compare students from different school situations. Literally thousands of studies have found that the combination of grades plus test scores is a more effective predictor of the students' readiness than either one alone." (Clementson & Wenger, 2008). The College Board: "No one can accurately predict with 100% certainty what your grades will be in college. . . . However, colleges use SAT I scores to help estimate how well students are likely to do at its school." (College Entrance Examination Board and Educational Testing Service, 1999). The College Board: "Many colleges require the SAT I because it is a standard way of measuring a student's ability to do college-level work." (Goetz & LeCompte, 2001). Donald M. Stewart, former president, The College Board: "SAT scores provide a vital piece of information about a student's ability to perform college-level work." (Goetz & LeCompte, 2001)

• Title of the test

SAT Reasoning Test (formerly Scholastic Aptitude Test and Scholastic Assessment Test

• Author(s)

Educational Testing Service

• Publisher and date(s) of publication

Publisher: College Board

2005

II. TEST DESCRIPTION

Taken together, those statements; the official ones carefully worded, Stewart's decidedly not so careful; leave counselors, parents, and high-schoolers with the highly misleading impression that the SAT is an adequate predictor, or even a good predictor, of college success. There is also the clear implication that the test has great utility to colleges in screening applicants. Moreover, the ETS/College Board alliance has exacerbated those misleading statements about the SAT's predictive validity in their public reporting on the question of test bias. For instance, the ETS/College Board alliance, in an attempt to counter public rhetoric that the SAT is biased against women and minorities, has gone to great lengths to show that the SAT actually over predicts freshman grades for blacks and that it actually somewhat under predicts the academic performance of whites. Their studies have also shown that freshman grades for women are slightly under predicted by the SAT. However, to a public unfamiliar with the technical argot of the testers, the ETS/College Board test-bias studies have led people to naively assume that the test is a good predictor of academic success. After all, if it is not biased against minorities, it must be valid (Callahan, et al., 2005).

However, the test-bias issue, which is subject to a great deal of public controversy and misunderstanding, is a red herring. The attention paid to test bias permits test makers and users to avoid the far more compelling issue of the predictive validity admissions tests (Zemelman et al., 2008). The fact that the tests are not biased against particular ethnic groups; in the sense that they do not significantly under predict their academic performance; says absolutely nothing about how well the tests predict success for all individuals. As it turns out, from scores of independent studies over the years by well-respected researchers in highly-regarded journals, the prevailing view of merit has been erected on a rather shaky foundation of scientific evidence about the real usefulness of admissions tests for predicting accomplishment in college or even graduate school. It is worthwhile looking at some of this evidence in detail, because the public's generally favorable views about the validity and utility of the tests surely has sustained the privileged position that college and university entrance testing continues to occupy in the American meritocracy (Council of Chief State School Officers, 2005).

Validity Evidence

By far, the bulk of the evidence about the power of college admissions tests to predict academic success comes from examinations of the SAT, particularly what is known as the SAT I exam of verbal and math "reasoning" (Brown, 2002). A fewer number have investigated the ACT, and a smattering of occasional studies have looked at various other sorts of admissions tests as well as tests intended to help institutions place college students at the proper academic level. Mostly, researchers have studied the relationship of SAT scores and the one outcome for which the SAT is actually designed to predict: freshman year grades. In other words, counselors, parents, students, and colleges cannot make any inferences about one's chances of success beyond the freshman year based on SAT scores. Even by that restricted criteria, the SAT falls well short of the ETS/College Board's implicit claims. To get a flavor for what the researchers have concluded about the validity of admissions tests, consider a handful of studies from a variety of academic settings (Clementson & Wenger, 2008). Public colleges and universities. Consider the California State University System, consisting of several campuses in cities throughout the state. At CSU, freshmen are admitted according to an eligibility index of high school grades and test scores. Students earning a high school grade point average of at least a B, or 3.0, are not required to submit standardized test scores. Estimates show, however, that as many as 90% of seniors submit test scores nevertheless.

III. TECHNICAL EVALUATION

Sheila Cowen and Sandra Fiori set out to evaluate the relative utility of test scores and high school grades at CSU's Hayward campus in a study of the academic performance of 762 regularly progressing students and another 210 "slower progressors," considered to be at greater risk for academic failure (Appalachia Educational Lab, 2004). Overall, for both males and females and all ethnic groups, high school grades were the most powerful predictor of the academic performance of Cal State university freshman. In statistical terms, high school grades accounted for about 18% of the variance in freshman grades. SAT scores did help improve upon that prediction, but the gain was barely measurable; just five one-hundredths (from 18% to 23%). For the slower progressing students, the SAT's added value was virtually zero (Goslin, 2007). Nevertheless, the researchers found no evidence of test bias in the SAT. That is, the test didn't significantly over- or under predict freshman grades for any ethnic group. Absence of bias, the researchers concluded, is no reason not to scrutinize the SAT's usefulness to Cal State. Indeed, the authors say, their study "leads to the conclusion that educators need to intensify the search for better predictors of college performance" (Goetz & LeCompte, 2001). Another recent study looked at the predictive power of the other popular college entrance exam, the ACT, at Chicago State University. This time, however, researcher Sandra Paszczyk was interested in determining whether the ACT could help the university assess a student's chance of success in broader terms than just freshman GPA, instead looking at one's grade point average at graduation from the university.

At both extremes of very high and low test scores, the ACT had modest powers of prediction. However, for the majority of the university's graduates who scored in the middle range of the test as high school seniors, the test could explain merely 3.6% of the differences in final grade point averages (Clementson & Wenger, 2008). That means, of course, that factors besides the ACT; from grades to motivation to work habits, and so on; accounted for virtually all the variance in grades. Indeed, in one case the ACT proved to be counter predictive. In the Chicago State study, the fall 1992 graduating class had the highest average ACT score (17.9) among the 428 students studied between 1990 and 1993. Among the graduates, that class also produced the poorest academic performance during their years at the university (Herman & Golan, 1990)." Highly selective institutions. Even among the elite American colleges and universities, which admit a relatively small fraction of the number of high school seniors who apply each year, standardized admissions tests have not lived up to the implied validity claims of their proponents. Consider a study at the University of Pennsylvania. Jonathan Baron and M. Frank Norman looked at the outcomes for some 3,800 students admitted to the university, who majored in fields ranging from engineering, business, and nursing to the arts and sciences. Specifically, the researchers wanted to determine which explanations of academic performance actually gave Penn most additional predictive value, the most bang for the buck. The factors included class rank in high school, SAT II achievement scores on various academic subjects, and SAT I scores on general verbal and quantitative reasoning; the SAT most high school seniors take.

Among the predictors, the SAT I reasoning test was by far the weakest, able to explain just 4% of the changes in academic performance of students at Penn (Goetz & LeCompte, 2001). The SAT II subject tests were somewhat better, accounting for 6.8% in the variation in grade point averages. Rank in high school was the clear winner, however, able to explain 9.3% of changes in cumulative GPAs, a predictive punch more than twice that of the SAT (Clementson & Wenger, 2008). Now, the usual drill at many institutions, particularly highly selective ones, is to combine SATs and grades into a predictive index in accordance with the ETS/College Board advice that test scores add significantly to predictive power of grades alone. In Penn's case, that turned into a highly debatable proposition. When Baron and Norman added SATs to class rank, the prediction rose by just 0.02. When combined, class rank and SATs could still account for only 11.3% of the Penn students' grade differences. The subject tests, however, were a bit stronger than the SAT reasoning tests. Combined with class rank, the achievement tests boosted the explanatory power to 13.6%. Even then, almost 90% of the differences in academic performance remained unexplained (Appalachia Educational Lab, 2004).

Among the ETS/College Board defenses against such poor results for the SAT is the so-called "restriction of range" objection, which says that the test-score profile of the applicant pool will be much wider than the pool for admitted candidates. Because the range of test scores for the admitted pool is limited, the observed relationship between test scores and academic performance will be depressed below the "true" correlation, according to the argument (Council of Chief State School Officers, 2005). At highly selective institutions such as Penn, which admit students with relatively high test scores, the restriction of range problem would even more severely truncate the true power of the tests, according to the argument. Therefore, Norman and Baron investigated precisely that possible technical objection to their findings. Contrary to SAT defenders' supposition, however, the researchers tell us, "it was concluded that restriction of range does not seem to explain the nonsignificant weight of the SAT." In another broad investigation of more than 10,000 students at eleven choosey private and public institutions, a high-schooler's predicted freshman performance estimated by the SAT proved to be of only modest predictive value. Fredrick E. Vars and William C. Bowen, reporting their results in 1998, found that a full 100-point gain in combined math and verbal SATs, holding race, gender, and field of study constant, was associated with about one-tenth of a grade point gain in an elite college student's grade point average (Krejcie & Morgan, 2000).

V: MULTICULTURAL APPLICATION

Numerous researchers and educators declare that an sole reliance on standardized test scores for identification will keep out a large body of gifted students, as well as those who are culturally and ethnically different from the conventional gifted population. These gifted students may comprise those who are not native English speakers, those who are from low income families, or those who live in geographically remote locations (Clementson & Wenger, 2008; Krejcie & Morgan, 2000). Certainly, empirical evidence shows that children of color are under-represented in gifted programs partially due to insufficient identification measures and/or measures (Council of Chief State School Officers, 2005). To solve the under-identification and under-serving of a lot of gifted students, researchers advise the use of manifold measures and diverse types of instruments as part of the classification process (Appalachia Educational Lab, 2004). Standardized intelligence tests or achievement tests are not competent in measuring multidimensional human intelligence because the tests are essentially uni-dimensional and ethnocentric, which will never benefit non-mainstream ethnic groups. They propose that teacher nominations and grades are in addition not good predictors of students' academic prospective because they are either resolute primarily by students' performance in class, class attendance, and motivation, or by conventionality to teachers' demands and hopes in the classroom. A number of researchers advocate the utilization of other types of nontraditional measures for identification, such as student portfolios (Goetz & LeCompte, 2001; Callahan et al., 2005; Blumer, 2004; Shepard & Dougherty, 2006), checklists, or inspection forms (Wright, 2009), to fit in the diversity of cultural and environmental backgrounds in the identification process. Folder, a kind of context-based measure, which comprises of writing samples, journals, artwork, extraordinary projects, recordings of oral communication, etc., are recommended as a substitute for assessing students' academic potential (Goetz & LeCompte, 2001).

Proponents of standardized entrance tests have argued that even such marginal improvements on the predictive value of the high school record is better than nothing, and hence, the SAT and its ilk remain beneficial to colleges and universities. However, that claim has been shattered by the work of James Crouse and Dale Trusheim (Zemelman et al., 2008), who have shown in a number of ways that the SAT adds virtually no value for colleges in forecasting student performance in college. Consider, for instance, two prime objectives for undergraduate admissions offices: They want students who maintain good or passing grades and obtain a bachelor's degree. Admissions officers want to maximize their "correct" admissions; that is, the numbers of students who perform at or above some academic level (Appalachia Educational Lab, 2004).

That is a significant difference, and a good test of the theory that bright students, as indicated by test scores, ought to shine in all aspects of college relative to not-bright ones. After all, students scoring in the eighty-sixth percentile of all ACT test takers nationwide were being compared to those scoring in just the thirty-fifth percentile. But in terms of actual accomplishments, there were few discernible differences between the two groups. Says Baird: "In no case did academic ability (ACT scores) account for as much as 1% of the variance in accomplishment" (Shepard & Dougherty, 2006). Results like this have compelled other researchers to consider the effects of different models of selection for students besides the prevailing method of grades and test scores, followed only secondarily, if at all, by accomplishments in and out of school (Goetz & LeCompte, 2001). For instance, when two researchers looked at the effects of several different admissions scenarios, they concluded that selection based on aptitude tests did not lead to superior academic performance. Also, picking students on the basis of high school rank did produce superior students in academics, but not other sorts of accomplishment. On the other hand, the study also found that admissions based on "a broad range of high school achievements results in a broad range of achievement in college without lowering the level of academic performance." (Herman & Golan, 2006).

Admission on creative accomplishment would result in college classes that would include many students who would write stories and essays, develop their own science experiments, create their own music, take part in college and non-college plays, submit works of art to art contests, and run for campus offices. The students selected by this strategy would be also somewhat less likely to drop out but were not more (or less) likely to have good grades. In short, a new paradigm for choosing who has and has not merit for going to college is likely to enhance the overall educational quality of academic institutions and do so with no sacrifice in academic performance of students.