Sociotechnical Harms of Algorithmic Systems Scoping a Taxonomy for Harm Reduction

2025-04-26 0 0 1.03MB 19 页 10玖币

侵权投诉

Sociotechnical Harms of Algorithmic Systems: Scoping a

Taxonomy for Harm Reduction

Renee Shelby

Google Research, JusTech Lab

Australian National University

San Francisco, CA, USA

Shalaleh Rismani

McGill University

Montreal, Canada

Kathryn Henne

Australian National University

Canberra, Australia

AJung Moon

McGill University

Montreal, Canada

Negar Rostamzadeh

Google Research

Montreal, Canada

Paul Nicholas

Google

San Francisco, CA, USA

N’Mah Yilla-Akbari

Google

Washington, D.C., USA

Jess Gallegos

Google Research

New York City, NY, USA

Andrew Smart

Google Research

San Francisco, CA, USA

Emilio Garcia

Google

New York City, NY, USA

Gurleen Virk

Google

San Diego, CA, USA

ABSTRACT

Understanding the landscape of potential harms from algorithmic

systems enables practitioners to better anticipate consequences of

the systems they build. It also supports the prospect of incorporat-

ing controls to help minimize harms that emerge from the interplay

of technologies and social and cultural dynamics. A growing body

of scholarship has identied a wide range of harms across dier-

ent algorithmic technologies. However, computing research and

practitioners lack a high level and synthesized overview of harms

from algorithmic systems. Based on a scoping review of computing

research (n=172), we present an applied taxonomy of sociotech-

nical harms to support a more systematic surfacing of potential

harms in algorithmic systems. The nal taxonomy builds on and

refers to existing taxonomies, classications, and terminologies.

Five major themes related to sociotechnical harms — representa-

tional, allocative, quality-of-service, interpersonal harms, and social

system/societal harms — and sub-themes are presented along with

a description of these categories. We conclude with a discussion of

challenges and opportunities for future research.

CCS CONCEPTS

•Social and professional topics

→

Computing / technology

policy;•General and reference →Evaluation.

KEYWORDS

harms, AI, machine learning, scoping review

Permission to make digital or hard copies of part or all of this work for personal or

classroom use is granted without fee provided that copies are not made or distributed

for prot or commercial advantage and that copies bear this notice and the full citation

on the rst page. Copyrights for third-party components of this work must be honored.

For all other uses, contact the owner/author(s).

ACM AIES 2023, August 9-11th 2023, Montreal, QC

ACM ISBN 979-8-4007-0231-0/23/08.

https://doi.org/10.1145/3600211.3604673

ACM Reference Format:

Renee Shelby, Shalaleh Rismani, Kathryn Henne, AJung Moon, Negar Ros-

tamzadeh, Paul Nicholas, N’Mah Yilla-Akbari, Jess Gallegos, Andrew Smart,

Emilio Garcia, and Gurleen Virk. 2023. Sociotechnical Harms of Algorithmic

Systems: Scoping a Taxonomy for Harm Reduction. In AAAI/ACM Con-

ference on AI, Ethics, and Society (AIES ’23), August 8–10, 2023, Montréal,

QC, Canada. ACM, New York, NY, USA, 19 pages. https://doi.org/10.1145/

3600211.3604673

1 INTRODUCTION

Harms from algorithmic systems — that is, the adverse lived ex-

periences resulting from a system’s deployment and operation

in the world — occur through the interplay of technical system

components and societal power dynamics [

]. This analysis con-

siders how these “harms (not bounded by the parameters of the

technical system)" can “travel through social systems (e.g., judicial

decisions, policy recommendations, interpersonal lived experience,

etc.)" [

151

, n.p.]. Computing research has traced how marginal-

ized communities — referring to communities that face structural

forms of social exclusion [

130

] — disproportionately experience

sociotechnical harms from algorithmic systems [

]. Such ex-

periences include, but are not limited to, the inequitable distribu-

tion of resources [

], hierarchical representations of people and

communities [

156

242

], disparate performance based on identity

categories [

143

], and the entrenchment of social and economic

inequalities [

]. In this way, algorithmic systems’ enactment

of power dynamics [

106

162

] can function as a minoritizing prac-

tice [64] through which unjust social hierarchies are reinforced.

Practitioners have sought to develop practices that better identify

and minimize sociotechnical harms from algorithmic systems (e.g.,

[

135

144

]). This includes work to taxonomize harms in HCI

on digital safety [

193

216

], in sociolegal studies on technology-

facilitated violence [

109

231

], and canonical responsible ML re-

search on representational, allocative [

], and quality-of-service

harms [

] that have signicantly shaped the responsible ML eld

arXiv:2210.05791v3 [cs.HC] 19 Jul 2023

ACM AIES 2023, August 9-11th 2023, Montreal, QC Shelby et al.

and standards development [

146

]. Alongside broader movements

towards regulation and standardization [

208

], harm reduction prac-

tices often draw on elds of auditing, impact assessment, risk man-

agement, and safety engineering where a clear understanding of

harm is essential [

112

]. Researchers have also developed “ethics

methods” [

150

] for practitioners to identify and mitigate sociotech-

nical harms, including statistical assessment [

135

145

], software

toolkits [

], and algorithmic impact assessments and audits [

178

providing notable benet in how harms are anticipated and identi-

ed within algorithmic systems. Existing work on dening, taxon-

omizing, and evaluating harms from algorithmic systems, however,

is vast and disparate, often focusing on particular notions of harm in

narrow circumstances. As such, it presents navigational challenges

for practitioners seeking to comprehensively evaluate a system

for potential harms [

178

179

182

], particularly for large gener-

ative models that perform dierent tasks across many use cases.

Moreover, the use of dierent terminologies for describing similar

types of harm undermines eective communication across dierent

stakeholder groups working on algorithmic systems [135, 159].

Recognizing these challenges, we conducted a scoping review

[

128

] and reexive thematic analysis [

] of literature on sociotech-

nical harms from algorithmic systems, oering a taxonomy to help

practitioners and researchers consider them more systematically. A

scoping review oers a generative starting place for a landscape

harm taxonomy. The purpose of a scoping review is to map the

state of a eld [

128

]; and here, provides a synthesis of existing

articulations of harm, calls attention to forms of harm that may

not be well-captured in regulatory frameworks, and reveals gaps

and opportunities for future research. As scholarly articulations

of harm emerge from dierent epistemic standpoints, values, and

methodologies, this paper pursues the broader question of: How

do computing researchers conceptualize harms in algorithmic sys-

tems? Three research questions guide this work:

(1)

What harms are described in previous research on algorith-

mic systems? How are these harms framed in terms of their

impacts across micro, meso, and macro levels of society?

What social dynamics and hierarchies do researchers of al-

gorithmic systems implicate in their descriptions of harms?

(2)

Where is there conceptual alignment on types of harm from

algorithmic systems? What type of organizational structure

of harms is suggested by conceptual alignment?

(3)

How do gaps or absences in research on sociotechnical harms

suggest opportunities for future research?

This research contributes to computing scholarship and respon-

sible AI communities, oering:

•

A scoping review of harms, creating an organized snapshot

of articulations of computational and contextual harms from

algorithmic systems;

•

A reexive thematic analysis of harms denitions, their im-

pacts to individuals, communities, and social systems, pro-

viding a framework for identifying harms when conducting

impact and risk assessments on an algorithmic system;

•

Support for interdisciplinary communication by providing

terms, denitions, examples of harms, and directions for

future work.

In what follows, we discuss the sociotechnical character of harms

from algorithmic systems and existing harm taxonomies, followed

by a description of our methodology (Section 3). We then detail the

harm taxonomy (Section 4), and propose next steps for related work

(Section 5). This analysis oers a starting place for practitioners

and researchers to reect on the myriad possible sociotechnical

harms from algorithmic systems, to support proactive surfacing

and harm reduction.

2 BACKGROUND

2.1 Sociotechnical Harms

Scholars in HCI, machine learning, Science and Technology Studies

(STS), and related disciplines have identied various harms from

digital technologies (e.g., [

121

192

193

207

]). This literature

underscores harm as a relational outcome of entangled dynamics

between design decisions, norms, and power [

148

238

particularly along intersecting axes of gender [

197

], race [

106

156

], and disability [

], among others. Harms from algorithmic

systems emerge through the interplay of technical systems and

social factors [

] and can encode systemic inequalities [

141

143

213

]. This duplicity of technology, as Ruha Benjamin [

]

explains, is a challenge: algorithms may have benecial uses, but

they often adopt the default norms, and power structures of society.

Recognizing the sociotechnical character of harms from algorith-

mic systems draws attention to how the development and experi-

ence of digital technologies cannot be separated from cultural and

social dynamics [

172

175

]. As van Es et al. [

224

, n.p.] note,

“algorithms and code reduce the complexity of the social world into

a set of abstract instructions on how to deal with data and inputs

coming from a messier reality.” This process involves design deci-

sions predicated on “selection, reduction, and categorization” [

]

through which technologies come to reect the values of certain

worldviews [

212

]. Without intentionally designing for equity,

algorithmic systems reinforce and amplify social inequalities [60].

2.1.1 Identifying and anticipating harms in practice. With increased

awareness of the need to anticipate harms early in product de-

velopment [

195

], designers and researchers are central actors in

pursuing harm reduction [

]. Anticipating harms requires

considering how technological aordances shape their use and

impact [

200

]. It can be done in relation to the technology holis-

tically or with a focus on certain features of the technology and

its use by dierent groups [

]. This work requires thinking criti-

cally about the distribution of benets and harms of algorithmic

systems [

169

] and existing social hierarchies [

]. It can be

strengthened by bringing in dierent standpoints and epistemolo-

gies, such as feminism [

155

], value-based design [

116

], de-

sign justice perspectives [

], and post-colonial theories [

126

148

Importantly, the process requires attending to the constitutive role

of social power in producing sociotechnical harms; “designers need

to identify and struggle with, alongside the ongoing conversations

about biases in data and code, to understand why algorithmic sys-

tems tend to become inaccurate, absurd, harmful, and oppressive” [

p. 2]. Thus, in anticipating harms, practitioners need to account for

computational harms as well as those arising through contextual

use [40, 164, 191, 236].

Sociotechnical Harms of Algorithmic Systems ACM AIES 2023, August 9-11th 2023, Montreal, QC

2.2 Taxonomies of Harm, Risk, and Failure

Structured frameworks can aid practitioners’ anticipation of harms

throughout the product lifecycle [

135

239

]. They encourage more

rigorous analysis of social and ethical considerations [

135

], espe-

cially when operationalizing principles seems opaque [

147

]. Taxon-

omizing harms is, however, an exercise in classication, which has

potential limitations: taxonomies can draw action to some issues

over others, shaping how people navigate and act on informa-

tion [

]. As such, the epistemological choices made in developing

harm taxonomies focus attention on certain areas over others [

Many existing harm taxonomies address particular domains of

use (e.g., [

193

217

]) and how they are complex assemblages of ac-

tors, norms, practices, and technical systems, which can foster indi-

vidual and collective harm [

174

]. Taxonomies have been developed

related to online content [

193

227

], social media [

157

217

218

users “at-risk" for experiencing online abuse [

216

234

], and mali-

cious uses of algorithmic systems [

], including cyber attacks [

]

and cyberbullying [

]. Relatedly, they can focus on particular types

of harms, such as misinformation [

218

] or representational harms

co-produced through image tagging system inputs and outputs

that reinforce social hierarchies [

121

228

]. While domain-specic

taxonomies draw attention to how context informs the emergent

nature of harm, they are not easily applicable to a wide range of

algorithmic systems. Many systems deploy across contexts, includ-

ing, for example, ranking and recommendation systems (e.g., search

engines or content sorting algorithms on social media platforms)

and object detection models (e.g., video surveillance systems, self-

driving cars, and accessibility technology).

Another common approach is to orient harm taxonomies around

specic algorithmic or model functions (e.g., [

235

]). Model-

focused taxonomies have been developed for large language mod-

els [

235

], image captioning systems [

121

228

], and so-called “foun-

dational models,” such as GPT-3 and BERT, which are applied in a

wide range of downstream tasks [

]. Organizing harm by model

function is highly useful when practitioners’ focus is on a singular

model because it draws attention to relevant computational con-

cerns. It does, however, pose limitations to practitioners working

downstream on products and features, where multiple model types

operate simultaneously, such as in social media, search engines,

and content moderation, and where contextual use signicantly

shapes potential harms.

Scholars have developed harm taxonomies related to system

misbehaviors and failures [

198

], particularly to aid algorithmic

auditing [

177

]. These taxonomies focus on how algorithmic sys-

tems are sources of harm (e.g., faulty input/outputs, limited testing,

proxy discrimination, and surveillance capitalism) [

203

]. Bandy [

]

summarizes four problematic behaviors of algorithmic systems —

discrimination, distortion, exploitation, and misjudgement. Using

such taxonomies focus attention to how specic aordances, train-

ing data, and design choices can co-produce harm [

]. Failure-

based taxonomies are helpful when practitioners examine potential

failure modes of a specic technology, but are often limited in terms

of helping to anticipate who or what is harmed.

In sum, taxonomies can be helpful tools for appreciating and

assessing how harms from algorithmic systems are sociotechnical.

As they retain social and technical elements, they cannot be reme-

died by technical xes [

] alone. They require social and cultural

change [

171

]. The proposed taxonomy provides a holistic and sys-

tematic view of the current discourse on types of sociotechnical

harms from algorithmic systems. As the scope of the taxonomy

we propose is broad, and many topic- or application- specic tax-

onomies exist already, we refer to and build on these existing works

when appropriate.

3 METHODOLOGY

In alignment with prior calls to anticipate computational and con-

textual harms [

151

], we synthesize insights on harms from

computing research to aid anticipation of sociotechnical harms

from algorithmic systems. Our ndings draw on a scoping review

for data collection and a reexive thematic analysis of computing

research on harms.

3.1 Overview of Methodology

Our approach followed prior scoping reviews in HCI literature [

223

], in alignment with the extension of the PRISMA checklist [

133

the PRISMA-ScR (Preferred Reporting Items for Systematic reviews

and Meta-Analyses extension for Scoping Reviews) [

220

]. Scoping

studies, as a knowledge synthesis methodology, map existing litera-

ture, and “clarify working denitions and conceptual boundaries of

a topic or eld” [

168

, p. 141]. They are especially appropriate when

distilling and sharing research ndings with practitioners [

221

and are suited to identifying evidence on a topic and presenting it vi-

sually. Compared to systematic reviews, scoping reviews address a

broader range of topics and incorporate dierent study designs [

A scoping review is an eective method for surfacing current “pri-

orities for research, clarifying concepts and denitions, providing

research frameworks or providing background, or contextual infor-

mation on phenomena or concepts" [

, p. 2104]. We implemented

a ve-stage scoping review framework [

128

]: (1) Identifying

research questions; (2) Identifying relevant studies; (3) Study se-

lection; (4) Charting the data; and (5) Collating, summarizing, and

reporting results.

3.1.1 Identify research questions. To identify the types and range

of sociotechnical harms, we developed the three aforementioned

research questions (see: Section 1).

3.1.2 Identify and select relevant studies. We then employed mul-

tiple strategies to identify relevant resources through dierent

sources: electronic scholarly databases, a citations-based review,

and targeted keyword searches in relevant organizations, and con-

ferences. Using the ACM Guide to Computing Literature as the

primary search system – which reects key computing research

databases – we developed the following initial set of key concepts to

search full text and metadata: “algorithmic harm”, “sociotechnical

harm”, “AI harm”, “ML harm”, “data harm”, “harm(s) taxonomy, “al-

locative harm”, and “representational harm.” Within scoping review

methodology, keyword search strategies should be devised to sur-

face relevant literature [

] and include terms common to the eld

[

101

]. We included allocative and representational harm as search

terms because of their conceptual dominance in machine learning

literature since 2017, popularized by responsible ML scholars (e.g.,

ACM AIES 2023, August 9-11th 2023, Montreal, QC Shelby et al.

[

]), enabling us to surface relevant literature in that sub-eld.

Iterative searching is a feature of scoping reviews [

]. Next, we

reviewed each paper, and conducted a citations-based review to sur-

face additional references (e.g., gray literature by nongovernmental

organizations (NGOs)) that materially discuss harms, but may not

use the specic terminology of the search terms. The citations-based

review revealed highly-cited cross-disciplinary scholarship (e.g., ar-

ticles from sociology, STS). Lastly, we relied on existing knowledge

and networks to surface additional sources, including IEEE, NIST,

Data and Society, the Aspen Institute, and the AI Incident Database.

Paper identication started February 2022. The initial search of

the ACM database produced a set of 85 research articles (duplicates

removed from 118 papers). The citations-based review and targeted

keyword searches of NGO and professional organization outputs

identied an additional 125 resources. We included articles that

described or discussed: (1) algorithmic technologies and (2) harms or

adverse impacts from algorithmic systems. We excluded 38 articles

that: (1) did not meet the inclusion criteria, and (2) did not have full-

text available. In total, 172 articles and frameworks were included

in our corpus (see: Appendix Figure 2 and Table 7).

3.1.3 Data charting. We employed a descriptive-analytical method

for charting data – a process of “synthesizing and interpreting qual-

itative data by sifting, charting and sorting material according to

key issues and themes" [

, p. 26]. Two researchers independently

charted the following data items extracted through reading of the

full text of each source and organized them into a spreadsheet:

(1) characteristics of sources: publication year and venue; and (2)

description of harm: denition or conceptual framing. Discovery of

a new concept or type of harm resulted in a new code, and repeat

encounters with existing concepts or harms were documented to

reach theoretical saturation – a point at which coding additional

papers or resources do not yield additional themes [

107

190

]. It

is dicult to know when to stop searching for new sources when

conducting a review [

101

]. Thus, relying on scoping reviews’ itera-

tive characteristic [

168

], we used theoretical saturation as a signal

to stop sourcing new papers. The entire corpus was coded. Data

charting concluded July 2022.

3.1.4 Collating and summarizing results. As collation of themes re-

quires synthesis and qualitative analysis of articles, we used Braun

and Clarke’s reexive thematic analysis [

], a post-positivist

approach to analysis that acknowledges researchers’ standpoints

inuence data interpretation and encourages self-reection. Within

reexive thematic analysis, coding is iterative as researchers are

immersed in the data. As coding is an evolving, self-reective pro-

cess in reexive thematic analysis, the authors engaged in deep

data immersion, interpretation, and discussion, including sharing

points of disagreement. First, we thematically sorted denitions

of code and looked at the frequency at which harm denitions

appeared to begin to identify dominant terms and denitions. Then,

we conducted a rst line-by-line pass reviewing associated phras-

ing and terminology of each specic kind of harm. In this initial

phase, we identied codes that could be easily condensed. For in-

stance, ‘physical harm’ and ‘physical injury’ were condensed into

one code: physical harm. We then began to cluster harms based on

the context or domain in which they were mentioned. For example,

specic harms describing forms of harassment (e.g., non-consensual

sharing of explicit images, or online stalking) were clustered under

an initial theme of “hate, harassment, and violence.” There were

many conceptual overlaps among harm types; denitions were not

always consistent. If there was a dominant term or denition in the

cluster that could encompass dierent sub-types of harm (based

on frequency of citation), that term was chosen as the primary

category. Notably, we identied and coded more than one type of

harm for approximately 80% of the articles in the corpus.

As RQ2 seeks to uncover where there was conceptual alignment

across computing sub-elds, initial decisions about harm type and

sub-type naming were made after raw coding the entire corpus and

discussing emergent themes. From this clustering, and as we iterate

from initial codes to nal themes, we developed a rst version of

the harm taxonomy. Three of these harm types — allocative, repre-

sentational, and quality-of-service — reect where there was strong

denitional and terminological consensus in pioneering responsible

ML literature (see: [

228

]). As we iterated from initial codes

to nal themes we chose to anchor to these canonical harm types

in alignment with the RQs. Social system harms and interpersonal

harms took shape through the collating and summarizing process.

Here, some of the sub-harm types, such as technology-facilitated

violence [

109

] and information harms [

232

] are existing and well-

established concepts/terms in dierent computing sub-elds to

which we anchored in alignment with RQ2. See the Appendix for

further details on the methodology and descriptive statistics of the

corpus.

In scoping reviews, collating and summarizing ndings requires

researchers to make choices about what they want to prioritize.

As the guiding purpose of this research was to develop an applied

taxonomy, we prioritized keeping the number of major categories

comprehensive yet manageable, envisioning a practitioner with

minimal knowledge of harms as the primary user. With the goal

of making the taxonomy accessible to practitioners with dierent

disciplinary backgrounds, we repeated this process of clustering

and synthesizing three times, rening language and examples of

harms to ensure clarity and conceptual cohesion.

Importantly, while we have aligned to canonical concepts we

found denitional variability within and across computing sub-

elds, illuminating how understandings of harm are not rigidly

xed and can shift based on sub-eld, context of use, technology

type, and the evolving state of knowledge. Our descriptions of

harm types and sub-types reect the rich variability that exists

in the broader eld and is not intended to usurp specic harm

denitions that hold specic meaning in dierent domains (e.g.,

law, engineering, policy, community work). As such this taxonomy

navigates the challenging task of synthesizing multidisciplinary

computing research with dierent priorities and concerns.

3.2 Limitations

In seeking to map how computing researchers conceptualize so-

ciotechnical harms in algorithmic systems, our scoping review

focused on academic outlets. The ndings are reective of existing

scholarly knowledge for a particular bounded time period. Like all

knowledge systems, computing research scholarship is not neutral;

it is shaped by various inuences, including researcher and insti-

tutional priorities, access to resources, thematic conferences, and

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

SociotechnicalHarmsofAlgorithmicSystems:ScopingaTaxonomyforHarmReductionReneeShelbyGoogleResearch,JusTechLabAustralianNationalUniversitySanFrancisco,CA,USAShalalehRismaniMcGillUniversityMontreal,CanadaKathrynHenneAustralianNationalUniversityCanberra,AustraliaAJungMoonMcGillUniversityMontreal,CanadaN...

展开>> 收起<<

Sociotechnical Harms of Algorithmic Systems Scoping a Taxonomy for Harm Reduction.pdf

共19页,预览4页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Sociotechnical Harms of Algorithmic Systems Scoping a Taxonomy for Harm Reduction

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: