discretionary and unpredictable.

carceral AI is used in inconsistent, discretionary, and unpredictable ways.

Carceral AI is often promoted as an “evidence-based” reform to remove subjectivity from human decision-making, but the use of such technology does not remove human input so much as obfuscate it. In practice, human decision-makers like judges, police officers, and probation officers differ widely in their adherence to algorithmic recommendations and follow them inconsistently in different contexts, leading to new sources of conflict and less clear accountability.

While practitioners hold up the objective backing that the algorithm provides, researchers have found that algorithmic recommendations often get applied selectively and in ways that can increase discrimination in practice. For example, judges using a sentencing risk assessment tool in Virginia provided harsher sentences for Black defendants than for white defendants who received the same risk score. In other cases, practitioners exhibit “algorithm aversion,” or resist engaging with carceral AI at all. Police officers have been found to ignore predictive policing hotspots in favor of their own judgment in an effort to reaffirm their decision-making authority, and judges and pretrial officers have been observed ignoring or overriding risk assessment recommendations. These powerful decision-makers express resentment to how the technology threatens their authority, and their resistance to its implementation can at times render it obsolete. While technology with minimal impact is better than technology with negative impact, a better alternative would be low-tech decarceration efforts that do not rely on individual decision-makers’ alignment with policy goals.

Carceral AI may also be used beyond its initial purpose in a process deemed “function creep,” such as being repurposed to surveil workers. Police agencies collect massive amounts of dragnet surveillance data without a clear plan for how it will be used or shared later on. Access to technology may also further reproduce existing power imbalances. Prosecutors have greater access to technology providers, data, and the experts doing analysis than public defenders, exacerbating an already prevalent imbalance in the use of surveillance technology, risk assessment instruments, and DNA profiling.

Finally, while the exact responses to technological intervention vary, its use consistently adds a level of opacity into the decision-making process and obscures accountability for the ultimate judgment. The “black box” nature of algorithms, particularly proprietary software whose source code is hidden from public scrutiny, makes it more difficult for defendants to contest life-altering decisions or pinpoint potential sources of error.