AI Decision-Making and Algorithmic Bias: Key Issues

Introduction: AI in High-Stakes Decision-Making

Artificial intelligence (AI) is increasingly being used in sensitive areas such as healthcare, the criminal justice system, and hiring processes. Courts today use AI systems to assess offenders' risk of recidivism and the likelihood of flight for offenders awaiting trial. One such algorithm is the Arnold Foundation algorithm, which draws on 1.5 million criminal cases to predict defendant behavior during pretrial hearings (Zavrsnik, 2020).

With such advancements, a growing debate has emerged over whether AI should fully replace human decision-making, given its perceived ability to eliminate human bias. Evidence suggests, however, that contrary to popular belief, AI decisions are not always less biased than human ones. A 2016 investigation by ProPublica found that data-driven AI used by courts to assess recidivism risk was biased against minorities and people of color (Silberg & Manyika, 2019). In the UK, a computer program used to determine which applicants would be invited to interview for medical school was found to be biased against female applicants and those with non-European names (Silberg & Manyika, 2019). This paper analyzes the potential causes of such biases in AI use and considers what decision-making is likely to look like in the future.

Potential Causes of Bias in AI Use

Several causes of algorithmic bias in AI systems have been identified. Bias can be introduced into data through pre-existing cultural, social, and institutional expectations that perpetuate historical or societal inequities (Silberg & Manyika, 2019). For instance, data may be compiled using language that reflects gender stereotypes embedded in society. A hiring algorithm designed to favor words such as "captured" or "executed" is more likely to be biased against female applicants, as such words appear more commonly in men's applications than in women's (Silberg & Manyika, 2019).

One algorithm that exhibited this kind of bias was developed by Amazon engineers to assist with the company's recruitment process. The algorithm was designed to recognize word patterns in applicants' résumés rather than relevant skills. The AI software penalized any résumé containing the word "women" and downgraded applicants who attended women's colleges, producing a clear gender bias (Lee, Resnick, & Barton, 2019). When historical biases are factored into training data, the resulting model will replicate the same kinds of flawed judgments that human decision-makers make.

Bias may also be introduced through the data collection techniques employed. For instance, a financial algorithm developed using sampling techniques that underrepresent certain minority groups could produce models with lower approval rates for those groups (Silberg & Manyika, 2019).

Another potential cause of AI bias is technical limitations in the design of the algorithm itself (Silberg & Manyika, 2019). Due to these limitations, an algorithm may identify statistical correlations that are either illegal or socially unacceptable. For example, a mortgage lending algorithm might find that older individuals carry a higher risk of default and consequently reduce lending as applicant age increases. If the algorithm recommends loans to younger applicants while denying them to older applicants on this basis alone — and this behavior is repeated across multiple cases — the algorithm would be considered biased against older loan applicants.

Bias may also arise when an algorithm is used in unanticipated contexts or with audiences not considered in its original design. In research by Sweeney and Latanya on racial differences in online advertising, searches for names common among African-Americans produced more advertisements containing the word "arrest" than searches for names common among white Americans (Silberg & Manyika, 2019). The researchers hypothesized that even if versions of the advertisement with and without the word "arrest" were initially displayed equally, users may have clicked on different versions more frequently depending on the search, leading the algorithm to display the arrest-linked version more often (Silberg & Manyika, 2017). In this case, the algorithm exhibits bias against African-Americans because it is deployed in a context other than the one for which it was designed — its intended use was marketing, not assessing arrest rates by race. Because algorithms respond to billions of user actions every day, this represents a significant and ongoing source of bias.

Conclusion

The use of AI in high-stakes decision-making domains remains promising but is complicated by persistent sources of algorithmic bias rooted in historical inequities, flawed data collection, technical limitations, and unanticipated deployment contexts. Until these causes are fully understood and addressed, a hybrid approach that combines human judgment with AI capabilities represents the most responsible path forward. This requires not only ongoing audits of AI systems but also a renewed commitment to examining and improving the quality of human decision-making itself.