What if assumptions of bias factored into test results to overcome social or cultural bias that prevents some people from achieving high test scores turned out to be flawed?

That's a messy sentence, right?   Confusing sentences like that are what happens when 40 years of accepted practice in using  tools to check tests of "general mental ability" for bias are themselves flawed. If it holds up, this finding from the Indiana University Kelley School of Business challenges basically throws out reliance on those exams to make objective decisions for employment or academic admissions.

"Test bias" means that two people with different ethnicity or gender, for example, who have the same test score are predicted to have different "scores" on the outcome (e.g., job performance); thus a biased test might benefit certain groups over others. Decades of earlier research consistently found no evidence of test bias regarding ethnic minorities, but the current study challenges this established belief.   Few topics have generated more public attention than bias in pre-employment and academic-admissions exams - as Asian men trying to get into California universities can attest, all scores are not equal.

"The irony is that for 40 years we have been trying to assess potential test bias with a biased procedure, and we now see that countless people may have been denied or given opportunities unfairly," says Herman Aguinis, professor of organizational behavior and human resources at Indiana University. "From an ethical standpoint it may be argued that even if only one individual is affected this way, that is one too many. The problem is obviously magnified when we are dealing with hundreds of thousands, if not millions, of individuals taking standardized tests every year."

The study, published in the July issue of the Journal of Applied Psychology, investigated an amalgam of scores representing a vast sample of commonly used tests, including civil service or other pre-employment exams and university entrance exams.  Aguinis led the study, which was co-authored by Steven A. Culpepper at the University of Colorado Denver and Charles A. Pierce at the University of Memphis.

To reach these conclusions, they created what they call the largest simulation of its kind (though it was still Monte Carlo), using nearly 16 million individual samples to yield more than eight trillion pairs of individual test/outcome scores. They built bias into most samples to resemble real-world results and checked tens of billions of scores. They found the procedures in use today overwhelmingly and repeatedly missed the bias inserted in the data.

Given the weight placed on such testing and the polarizing nature of the underlying racial/ethnic achievement gap, the authors expect their study will spur considerable controversy among the public and the academic, legal and policy communities, all of which will question the long-held belief that tests are unbiased.

They also anticipate a significant impact on the multi-billion dollar testing industry but made clear that they are not saying that any organization is deliberately using biased tests. However, as a preliminary step while more research is conducted, it is likely that many organizations will examine their existing tests and perhaps create new ones.

Citation: Aguinis, Herman; Culpepper, Steven A.; Pierce, Charles A., 'Revival of test bias research in preemployment testing', Journal of Applied Psychology, Vol 95(4), Jul 2010, 648-680. doi: 10.1037/a0018714