Abstract
Algorithmic accountability law—focused on the regulation of data-driven systems like artificial intelligence (AI) or automated decision-making (ADM) tools—is the subject of lively policy debates, heated advocacy, and mainstream media attention. Concerns have moved beyond data protection and individual due process to encompass a broader range of group-level harms such as discrimination and modes of democratic participation. While a welcome and long overdue shift, the current discourse ignores systems like databases, which are viewed as technically “rudimentary” and often siloed from regulatory scrutiny and public attention. Additionally, burgeoning regulatory proposals like algorithmic impact assessments are not structured to surface important –yet often overlooked –social, organizational, and political economy contexts that are critical to evaluating the practical functions and outcomes of technological systems.
This Article presents a new categorical lens and analytical framework that aims to address and overcome these limitations. “Suspect Development Systems” (SDS) refers to: (1) information technologies used by government and private actors, (2) to manage vague or often immeasurable social risk based on presumed or real social conditions (e.g. violence, corruption, substance abuse), (3) that subject targeted individuals or groups to greater suspicion, differential treatment, and more punitive and exclusionary outcomes. This framework includes some of the most recent and egregious examples of data-driven tools (such as predictive policing or risk assessments), but critically, it is also inclusive of a broader range of database systems that are currently at the margins of technology policy discourse. By examining the use of various criminal intelligence databases in India, the United Kingdom, and the United States, we developed a framework of five categories of features (technical, legal, political economy, organizational, and social) that together and separately influence how these technologies function in practice, the ways they are used, and the outcomes they produce. We then apply this analytical framework to welfare system databases, universal or ID number databases, and citizenship databases to demonstrate the value of this framework in both identifying and evaluating emergent or under-examined technologies in other sensitive social domains.
Suspect Development Systems is an intervention in legal scholarship and practice, as it provides a much-needed definitional and analytical framework for understanding an ever-evolving ecosystem of technologies embedded and employed in modern governance. Our analysis also helps redirect attention toward important yet often under-examined contexts, conditions, and consequences that are pertinent to the development of meaningful legislative or regulatory interventions in the field of algorithmic accountability. The cross-jurisdictional evidence put forth across this Article illuminates the value of examining commonalities between the Global North and South to inform our understanding of how seemingly disparate technologies and contexts are in fact coaxial, which is the basis for building more global solidarity.
Recommended Citation
Rashida Richardson & Amba Kak,
Suspect Development Systems: Databasing Marginality and Enforcing Discipline,
55
U. Mich. J. L. Reform
813
(2022).
Available at:
https://repository.law.umich.edu/mjlr/vol55/iss4/4
Included in
Comparative and Foreign Law Commons, Privacy Law Commons, Science and Technology Law Commons, Social Control, Law, Crime, and Deviance Commons