Durham Research Methods Conversations - Interdisciplinary Interrogation of Sociotechnical Systems

This blog post is on a conversation I was steering for the Research Methods Conversations. (I feel like I have to excuse myself for it, as I wrote it with little time on my hands…)

In the beginning of March, the Research Methods Cafe hosted Javier S. Monedero, who is a Distinguished Researcher “Beatriz Galindo” at the AYRNA research group in the Dept. of Computer Science at the University of Córdoba, joined the conversation to share his experience of working in an, for his background, unusual research environment. As a former research asssociate and affiliate researcher with the Data Justice Lab (DJL; Cardiff), he is bridging the social and communication sciences with computer sciences, working in the knowledge gaps separating them. On such an endeavour, one faces many obstacles that need to be overcome to ensure integrity of information in the presence of diversity. Intrigued to find out more, the conversation was joined by a large group of students and scholars from many departments, including Anthropology, Sociology, Sport & Exercise, Education, Government and International Affairs, University Library and Collections, Business School, and Physics.

Javier started off with a presentation, in which he sketched out a possible framework that seeks to ensure an ethicopolitically aware manner in which data is made actionable. It applies to many data driven algorithms that support the governance of our everyday lives, such as, the regulation of social protection schemes (e.g. in India and Australia), international development or the environment. The framework takes a holistic approach, incorporating and respecting the plurality of various actors, interests and social forces that shape how and on what terms society is increasingly being datafied. In this way, it addresses issues relating to democratic procedures, the entrenchment and introduction of inequalities, discrimination and exclusion of certain groups, deteriorating working conditions, or the dehumanisation of decision-making that have been brought about by the drive for automatisation (as exemplified recently in the Netherlands).

Following the layout of an ideal came the reflection on current practices and narratives that try to mitigate the above mentioned issues. For it he referred to his work on COMPAS, a data driven algorithm that assesses the likelihood of a defendant becoming a recidivist that was used in the US criminal justice system. Following the investigation by ProPublica, COMPAS has become one of the most documented and studied cases when it comes to assessing the impact of sociotechnical practices.

Noticeable of the assessments is that they are carried out through the lens of performance metrics and are concerned about privacy and data protection. This, however, encounters the following problems. Firstly, there is the difficulty of choosing against which metric an algorithm is being optimized. Traditionally the optimization of accuracy, sensitivity and specificity have been pursued with fairness being added only recently. If fairness is of concern, then one has to realize that it is near to impossible to produce a (general) statistical definition of fairness (due to the different and sometimes incompatible measures of fairness, e.g., distributive vs procedural fairness). Secondly, the implications of datafication go beyond individual privacy, since data processes are not ‘flat’ and do not implicate everyone in the same way. They entail power dynamics that require investigation and critique that go beyond technical questions but relate to long-standing social, political, economic and cultural issues. Therefore, if one seeks to assess the ethicopolitical effects of sociotechnical practices on our society, multiple entry-points from a range of scholarly disciplines are required.

However, working in a diverse research team that cooperatively finds those entry-points comes with its challenges. Each participant is often an expert in a highly specialized field, having a distinct perspective. To meet the scientific goals, S. L. Star and J. R. Griesemer (1989) identify the following requirements for collective work:

• develop, teach, and enforce a clear set of methods to ‘discipline’ information flow
• generate a set of boundary objects which maximise both the autonomy and communication between participants. To identify the optimal information flow and boundary object needs effort and doesn’t need to be static in time. Javier vividly remembers how long it can take to arrive at a common understanding and vocabulary, without which communication becomes impossible, i.e., information can’t flow.

Upon reflection by the participants on their own working environments, it was becoming apparent that the current academic framework makes it difficult to pursue and motivate highly interdisciplinary research efforts, even though they are of such importance. As jobs are often found within one particular department and not across them, people whose work does not align with one but multiple disciplines are less likely to find a faculty position. Whether this is the cause or not isn’t certain, but many questions and mistakes are repeated with new sets of tools that might be helped and avoided if information would flow more across departments. One can only imagine what impact it would have had, if the company behind COMPAS would have hired scholars from science and technology studies…