When most people think of junk science, they picture test tubes and laboratories. But not all junk science emanates from the stuff that comes out of Petri dishes or Erlenmeyer flasks. In this case, I’m talking about a research report that claims the SAT is biased against minorities, particularly African-Americans.
This report, authored by Maria Veronica Santelices with Pontificia Universidad Católica de Chile and Mark Wilson from the University of California at Berkeley, ostensibly looked at the results of SAT tests, examined which kids got which questions right and which questions wrong, and concluded that African American students didn’t do as well on the test as white students because the SAT people used certain words that minorities have a hard time understanding.
The Santelices/Wilson report was published by the Harvard Educational Review in April and idled around the Internet for a couple of months until Washington Post blogger Jay Mathews decided to write it up for his blog. His post revealed shocking details and ramifications involving the SAT:
“The confirmation of unfair test results throws into question the validity of the test and, consequently, all decisions based on its results,” said Maria Veronica Santelices, now at the Pontificia Universidad Catolica de Chile in Santiago, and Mark Wilson of UC Berkeley. “All admissions decisions based exclusively or predominantly on SAT performance–and therefore access to higher education institutions and subsequent job placement and professional success–appear to be biased against the African American minority group and could be exposed to legal challenge.”
Legal challenges to college admissions decisions! Could anything be more dire? As a matter of fact, things could be much more dire because this report is nothing more than junk science masquerading as scholarship.
A quick read of the Santelices/Wilson report shows quite a number of things that somehow escaped Mathews. It will cost you ten bucks to read the thing but if you’re really serious about looking for a case of badly done research, it’s money well spent.
For starters, the research examined four separate SAT tests, two of which were administered in 1994 and two of which were administered in 1999.
Those dates, by the way, are not typos. The tests really do date back to the Clinton Administration.
An examination of the performance of different groups of students on individual questions – something researchers refer to as differential item functioning or DIF – showed that in three of the four tests included in the study, there was no significant DIF between white students and minority students. But on one of the tests from 1999, researchers found a significant DIF on two questions.
Again, not a typo. Two questions.
Another salient fact that did not occur to Mr. Mathews is that the 1999 test with a significant DIF on two questions included the exact same questions as the second 1999 test that had no significant DIF – the questions were just presented in a different order. Rather than an indictment of the SAT, we have a statistical quirk!
But wait, there’s more.
Millions of kids across the country take the SAT every year as they begin their own personal Paper Chase in earnest. But this study didn’t look at that. It looked only at students from California and even then, it included only kids who were accepted and admitted to the University of California system. Wouldn’t a larger sample be needed to draw any meaningful conclusions?
We’re still not finished here –
The four tests included in the Santelices/Wilson study don’t exist anymore. That’s because the SAT went through a major overhaul in 2005 when it added a writing section and excised analogy questions.
Then there’s the issue of colleges being exposed to legal action for “admissions decisions based exclusively or predominantly on SAT performance.” I don’t know what college they might be referring to but here in the USA, there isn’t a single college that decides whom to admit based “exclusively or predominantly” on test scores. It just doesn’t happen.
When you go through the process of getting in to college, admissions officers look over everything – high school grades, class ranking, letters from teachers and guidance counselors, extracurricular activities, awards, entrance essay, community projects – all sorts of stuff. I’m no lawyer and I suppose it’s possible that a college could be in legal hot water for basing their admissions decisions on a single test score, but the fact is no college does that.
The Harvard Educational Review is not even a peer-reviewed journal. It’s run by grad students at Harvard. That’s probably why this study was published in this particular magazine. I can’t imaging any peer-reviewed journal ever accepting or publishing it because even a JunkScienceMom can read it and the conclusions simply don’t withstand scrutiny.
Timing matters too. This report was published more than two months ago but it’s taken until now for anybody in the media to pick-up on it. Why might that be? Probably because the notion of the SAT or any other college entrance exam being racist has been long disproved and debunked by serious researchers. Here’s what University of Minnesota researchers Paul Sackett, Matthew Borneman and Brian Connelly noted about the racial-bias theory:
This assertion consists of two parts. The first is that mean scores are lower for certain minority groups. This has long been known to be true; we offer details below. The second is that these mean differences can be interpreted as evidence of bias in the tests; this inference is unequivocally rejected within mainstream psychology.
So while mainstream psychology unequivocally rejects the shopworn racial bias theory – which is why the Santelices/Wilson study languished for months with exactly zero media attention – Jay Mathews decides to give this disproved idea fresh legs.
What sort of legs is Jay ‘Class Struggle’ Mathews giving this deeply flawed report and its equally flawed theory? Here are some of the comments on the Washington Post website responding to the Mathews piece:
I’m a black man, with a black daughter, and I just hope she doesn’t read this crap. She will be taking the SAT next year, and I expect her to be highly competitive, period.
I find it all extremely unlikely. I wish you had quoted some of the key points.
We constantly hear about bias on tests – the New Haven firefighters’ exam springs to mind – but I have yet to see one question that has been pointed to as biased, much less one where someone can explain just how it is biased.
With all the problems swirling around this report by Santelices and Wilson it’s hard to figure out why anybody would bother to publish it much less report on it unless that reason is unadulterated bias. It’s the only thing that makes sense and it’s one form of bias that’s been proved real.
In fairness, I did find one line with which I could agree in the Mathews blog post on the study by Santelices and Wilson:
They are discussing a complex topic, full of psychometric terms and concepts I am not competent to judge.
One would hope that a writer for a newspaper like the Washington Post would be able to read through and understand the subject matter on which he’s writing, but Mathews admits that’s not the case with his latest piece of work. How can this pass for journalism?
Perhaps we will soon be graced by another insightful piece from some other writer about how the Earth is flat and the evil geniuses at the SAT have somehow tricked us into believing it’s spherical. Given this shoddy treatment of an equally shoddy piece of junk science, it’s not out of the realm of possibility.
COMMENTS
Please let us know if you're having issues with commenting.