We are living in a historical time when every little detail of our experience is turned into a data point that is used by AI systems to profile and make automated decisions about our lives. Increasingly more these technologies are used worldwide.
Health and education practitioners use them to ‘track risk factors’ or to find ‘personalized solutions’. Employers, banks, and insurers use them to judge clients or potential candidates. Even governments, the police and immigration officials use these technologies to make decisions about individual lives, from one’s right to asylum to one’s likelihood to commit a crime. The COVID-19 pandemic has only intensified and exacerbated these practices of technological surveillance, algorithmic profiling and automated decision making.
In different sections of society algorithmic profiling is often understood as holding the key to human nature and behavior; it is used to make the process of decision making more efficient, and to ‘avoid the human error’. Paradoxically, however – as recent research has shown – these technologies are filled with systemic ‘errors’, ‘biases’ and ‘inaccuracies’ when it comes to human profiling.
Of course, AI systems can bring much positive outcomes and this is clear if we consider their use in tackling specific issues such as diseases or climate change. Yet, when it comes to human profiling these technologies cannot grasp the complexity of human experience and behaviors, and their errors can have a real impact on individual lives and human rights.
In 2020, we launched The Human Error Project: AI, Human Rights, and the Conflict Over Algorithmic Profiling, because we believed that – in a world where algorithmic profiling of humans is so widespread – critical attention needs to be paid on how institutions, businesses, and individuals coexist, negotiate and construct meaning out of AI errors. The project remained active until 2025, providing both academic and knowledge-exchange results. This website serves as an archive of the project’s outcomes and as a historical reference of its activities.
In our research we use the term ‘the human error of AI’ as an umbrella concept to shed light on different aspects of algorithmic fallacy when it comes to human profiling: Bias – AI systems are human made and will always be shaped by the cultural values and beliefs of the humans and societies that created them. Inaccuracy – AI systems process data. Yet the data processed by algorithms is often the product of everyday human practices, which are messy, contradictory and taken out of context, hence algorithmic predictions are filled with inaccuracies, partial truths and mis-representations. Un-accountability – AI systems lead to specific predictions that are often unexplainable, and unaccountable. How can we trust or challenge their decisions, if we cannot explain or verify them? The combination of bias, inaccuracy, and lack of transparency in algorithmic predictions, we believe, implies that AI systems are often (if not always) somehow fallacious in reading humans.
The Human Error Project thus shared much of the same understandings of current research in the field of critical AI and data studies that has shown how AI systems are often shaped by systemic inequalities (Eubanks, 2018; Amoore, 2020; Crawford, 2021), by racial biases (Noble, 2018; Benjamin, 2019; Richardson et al. 2019; Atanasoski and Vora, 2019; Amaro, 2021) and by inaccurate and human reductionist analyses of human practices and intentions (Barassi, 2020; Milan, 2020).
Yet we also wanted to push the debate further and ask “What next?” The project set out to question what happens when different actors in society realise that AI systems can be fallacious and biased in reading humans, and when they discover that AI systems too can be racist, sexist, ageist, ableist and so on. We examined how different sections of society understood and shaped the political debate on the Human Error of AI, and how they negotiated and coexisted with the human rights implications of AI. We also looked at what solutions and AI futures different actors were envisaging.
We launched The Human Error Project because we believed that one of the most fundamental questions of our times has become that of mapping, studying, and analyzing the emerging debates and conflicts over AI errors and algorithmic profiling. With this project we positioned ourselves amongst those scholars that have called for an analysis of the ‘political life of technological errors’ (Aradau and Blanke, 2021) and for a qualitative approach to the understanding of algorithmic failures (Munk et. al, 2022; Rettberg, 2022).
Our aim has been to map the discourses and listen to the human stories of different sections of society, to try and understand how AI errors – when it comes to the profiling of humans – are experienced, understood and negotiated. The Human Error Project Team has researched three different areas of society where these conflicts over algorithmic profiling are being played out in Europe: the media and journalists; civil society organizations and critical tech entrepreneurs. For all these different sections of society we gathered data primarily through three main methodologies: critical discourse analysis, organizational mapping, and the collection of 100 in-depth interviews.
Our methodological approach has been based on the understanding that whilst most of current research and influential journalism in the field of critical AI studies comes from the U.S. and focuses on algorithmic injustice with reference mostly to U.S.-centric systems of inequality, European countries (within and outside the E.U.) and their cultural specificity are an equally interesting field of analysis for studying the ways in which the debate on AI errors, algorithmic profiling and human rights is being shaped. The project has been built on the understanding of AI errors as a top priority of our times, because they shed light on the fact that the race for AI innovation is often shaped by stereotypical and reductionist understandings of human nature, and by new emerging conflicts about what it means to be human.
The Human Error Project team has worked on different interconnected research streams:
Prof. Veronica Barassi has served as the Principal Investigator of the project (2020-2025), which has been funded by the University of St. Gallen’s Basic Research Fund (Grundlagenforschungsfonds GFF).
Dr. Antje Scharenberg has worked (2020-2022) on postdoctoral research project investigating the challenges of algorithmic profiling for human agency and led the work package dedicated to civil society.
Dr. Philip Di Salvo has worked (2022-2025) on a postdoctoral research project dealing with journalists covering issues of AI errors and algorithmic profiling and led the work package dealing with journalism.
Dr. Rahi Patra completed her PhD project (2020 – 2025) “Reimagining Technoscience from the Margins: Caste, Sanitation Work, and the Making of Inclusive Technological Futures” and contributed to the work package dedicated to entrepreneurs.
Dr. Marie Poux-Berthe completed her PhD project (2020-2024) “On Digital Ageism and Beyond – How Older Adults Unfold Aging and Technology Relations in France” and contributed to the work package dedicated to entrepreneurs.