From plane crashes to algorithmic harm applicability of safety engineering frameworks for responsible ML_2

2025-04-27 0 0 750.24KB 25 页 10玖币
侵权投诉
From plane crashes to algorithmic harm: applicability of
safety engineering frameworks for responsible ML
SHALALEH RISMANI, Google Research, McGill University, Canada
RENEE SHELBY, Google, JusTech Lab, Australian National University, U.S.A
ANDREW SMART, Google Research, U.S.A
EDGAR JATHO, Naval Postgraduate School, U.S.A
JOSH A. KROLL, Naval Postgraduate School, U.S.A
AJUNG MOON, McGill University, Canada
NEGAR ROSTAMZADEH, Google Research, Canada
Inappropriate design and deployment of machine learning (ML) systems leads to negative downstream social
and ethical impact – described here as social and ethical risks – for users, society and the environment. Despite
the growing need to regulate ML systems, current processes for assessing and mitigating risks are disjointed
and inconsistent. We interviewed 30 industry practitioners on their current social and ethical risk management
practices, and collected their rst reactions on adapting safety engineering frameworks into their practice
– namely, System Theoretic Process Analysis (STPA) and Failure Mode and Eects Analysis (FMEA). Our
ndings suggest STPA/FMEA can provide appropriate structure toward social and ethical risk assessment
and mitigation processes. However, we also nd nontrivial challenges in integrating such frameworks in
the fast-paced culture of the ML industry. We call on the ML research community to strengthen existing
frameworks and assess their ecacy, ensuring that ML systems are safer for all people.
CCS Concepts:
•Social and professional topics →Computing / technology policy
;
•General and
reference →Evaluation;Surveys and overviews.
Additional Key Words and Phrases: empirical study, safety engineering, machine learning, social and ethical
risk
ACM Reference Format:
Shalaleh Rismani, Renee Shelby, Andrew Smart, Edgar Jatho, Josh A. Kroll, AJung Moon, and Negar Ros-
tamzadeh. 2023. From plane crashes to algorithmic harm: applicability of safety engineering frameworks for
responsible ML. 1, 1 (October 2023), 25 pages. https://doi.org/XXXXXXX.XXXXXXX
1 INTRODUCTION
During a panel at the 1994 ACM Conference on Human Factors in Computing Systems (CHI),
prominent scholars from dierent disciplines convened to discuss "what makes a good computer
system good." Panelists highlighted considerations for safety, ethics, user perspectives, and societal
structures as critical elements for making a good system [
41
]. Almost 28 years later, we posit that
Authors’ addresses: Shalaleh Rismani, Google Research, McGill University, Montreal, Canada; Renee Shelby, Google, JusTech
Lab, Australian National University, San Francisco, U.S.A; Andrew Smart, Google Research, San Francisco, U.S.A; Edgar
Jatho, Naval Postgraduate School, Monterey, U.S.A; Josh A. Kroll, Naval Postgraduate School, Monterey, U.S.A; AJung Moon,
McGill University, Montreal, Canada; Negar Rostamzadeh, Google Research, Montreal, Canada.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and
the full citation on the rst page. Copyrights for components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior specic permission and/or a fee. Request permissions from permissions@acm.org.
©2023 Association for Computing Machinery.
XXXX-XXXX/2023/10-ART $15.00
https://doi.org/XXXXXXX.XXXXXXX
, Vol. 1, No. 1, Article . Publication date: October 2023.
arXiv:2210.03535v1 [cs.HC] 6 Oct 2022
2 Rismani et al.
these epistemological perspectives need to be in deeper conversation for designing and assessing
machine learning (ML) systems that challenge conventional understanding of safety and harm.
The development and use of ML systems can adversely impact people, communities, and society
at large [
14
,
32
,
82
,
89
,
118
,
125
], including inequitable resource allocation [
6
,
22
,
106
], perpetuating
normative narratives about people and social groups [
56
,
119
], and the entrenchment of social
inequalities [
4
,
70
,
76
]. We frame these adverse impacts broadly as social and ethical risks. To manage
such risks, quantitative [
38
,
66
], qualitative [
44
,
71
,
81
,
97
], and epistemological frameworks [
31
,
46
,
82
] have been proposed. In parallel, active regulatory and standards activities are taking place
internationally [
1
–
3
,
35
,
45
]. Despite the rapidly evolving discourse, there is limited empirical
understanding of how proposed social and ethical risk management tools have been adopted by
practitioners [36, 74].
Inspired by the 1994 panelists, we examine the dialogue between safety engineering frameworks
and understandings of social and ethical risks of
ML
systems. First, we report on ethical and
social risk management practices currently used in the industry. Second, we take a developmental
approach to examine how safety engineering frameworks can improve existing practices. We
chose two of the most successful safety engineering frameworks used in other sociotechnical
domains [
19
,
93
,
117
]: Failure Mode and Eect Analysis (FMEA) [
21
] and System Theoretic Process
Analysis (STPA) [67, 91], which we describe in detail in Section 2.
We conducted 30 semi-structured in-depth interviews with industry practitioners who shared
their current practices used to assess and mitigate social and ethical risks. We introduced the two
safety engineering frameworks, inviting them to envision how they might employ them to assess
ethical and social risk of ML systems. The results of our study address the following research
questions:
•RQ1: Which practices do ML practitioners use to manage social and ethical risks today?
•RQ2
: What challenges do practitioners face in their attempts to manage social and ethical
risks?
•RQ3
: How could safety engineering frameworks such as FMEA and STPA inform and improve
current practices? What advantages and disadvantages of each method do ML practitioners
identify?
We contribute to the emerging research on managing social and ethical risk of
ML
systems in
human-computing scholarship and responsible ML communities by oering:
•An overview of how practitioners dene, assess and mitigate social and ethical risks;
•An analysis of the corresponding challenges when implementing these practices;
•
A set of insights on how FMEA and STPA could inform existing practices along with their
perceived advantages and disadvantages;
•Future research directions and calls to action for HCI and responsible ML scholars.
Our ndings illustrate safety engineering frameworks provide valuable structure for investigating
how social and ethical risks emerge from ML systems design and integration in a given context.
However, successful adaptation of these frameworks requires solutions to existing organizational
challenges for operationalizing formal risk management practices. Moreover, results of our work
motivate further theoretical and applied research on adaptation of such frameworks. The remainder
of this paper is organized as follows. We start by providing an overview of current discourse in
responsible ML development and contextualize the relevance of the safety engineering frameworks
(Section 2). We outline our interview protocol and analysis methods in Section 3, followed by
highlighting key ndings in Section 4. We discuss the value and shortcomings of applying safety
engineering frameworks in light of current practices and call on the research community to further
, Vol. 1, No. 1, Article . Publication date: October 2023.
From plane crashes to algorithmic harm 3
examine and strengthen these frameworks for ethical and social risk management of
ML
systems
in Section 5.
2 BACKGROUND
Analyzing social and ethical implications of algorithmic systems is not new to computing researchers
and practitioners [
11
,
29
,
42
,
90
]. In the literature, terms such as harm [
118
], failure [
96
], and
risk [
1
,
125
] are often used to describe adverse impacts of ML systems. While there is currently no
agreed upon denition of these terms and their relationships, we use the phrase social and ethical
risk to frame broadly the adverse social and ethical implications ML systems can have on users,
society, and the environment. This working denition provides conceptual consistency in this
paper, and is not meant to be normative. In the remainder of this section, we contextualize current
discourses on social and ethical risks in ML to situate our study design, ndings, and discussion.
We highlight current epistemological perspectives and tools for responsible
ML
development, and
detail the safety engineering frameworks (FMEA and STPA).
2.1 Epistemological perspectives for anticipating and mitigating harms of ML systems
Scholars have proposed various methods for anticipating social and ethical impacts [
36
,
37
,
109
].
Anticipating harm involves thinking about the values [
88
,
111
] and aordances of ML systems [
17
],
with specic attention to how social norms and power dynamics constitutively shape adverse
impacts of ML systems [
12
,
13
]. The process of anticipation is aided by critical epistemologies that
center the needs and standpoints of socially oppressed groups, including critical race theory [
12
,
46, 58, 89], post-colonial theories [82], and queer [114] and feminist HCI [9].
As social and ethical impacts are co-constituted through the interplay of technical system com-
ponents and the social world [
53
], design methodologies attentive to these dynamics support more
meaningful harms anticipation and mitigation. For instance, Value Sensitive Design that examines
what value tensions ML systems create or resolve [
40
,
124
], supports increased stakeholder coordi-
nation [
121
] and consideration of technology from dierent social standpoints and perspectives
[
8
]. Similarly, participatory design methods can center the needs of users, communities, and other
stakeholders often excluded from the design process [
130
] or algorithmic governance [
64
,
65
],
especially when incorporating feminist epistemologies [
9
,
49
]. Speculative design can also help
designers imagine more socially just and racially equitable technological futures [
47
]. While these
epistemological perspectives and frameworks do not explicitly assess risk, they provide theoretical
grounds for examining and mitigating social and ethical risk.
2.2 Responsible ML tools, processes, and emerging regulations
With increased deployment of ML systems and reported harms [
14
,
89
,
125
], there is movement
towards formalizing quantitative and qualitative tools for responsible
ML
development. Tradi-
tionally,
ML
system evaluations [
50
,
104
] prioritized assessing and optimizing for a narrow set
of performance metrics, mistakenly treating these measurements (e.g., accuracy of a test set)
as a target rather than proxy for certain risks [
72
]. Recognizing these shortcomings [
55
],
ML
scholars proposed alternative methods to enable more comprehensive evaluation. These methods
include assessing computational fairness with alternative statistical denitions [
20
,
24
,
26
,
27
],
quantifying model interpretability based on statistical properties [
83
,
95
], evaluating robustness
to distribution shift [
23
,
59
,
116
] and examining model performance when exposed to adversarial
examples [34, 105, 128, 129].
In parallel, signicant eort has also focused on developing mixed-method (qualitative and
quantitative) processes to increase accountability and assess ML systems contextually. Scholars
have proposed model cards [
81
], datasheets [
44
] and auditing tools [
18
,
97
,
110
] to improve the
, Vol. 1, No. 1, Article . Publication date: October 2023.
4 Rismani et al.
transparency and quality of model and data practices. Human right and algorithmic impact assess-
ments aid identication of potential societal level harms by examining model deployment in a given
context [
63
,
77
,
85
]. Similarly, scholars have developed contextual methods of assessing fairness by
focusing attention on the situated power dynamics of where systems are deployed [
107
,
127
] and
transparency [
123
] of
ML
systems. Parallel to technique development, there is a rapidly emerging
set of international standards [
3
], policies [
120
], and regulatory frameworks [
35
] that examine ML
systems from a risk-based perspective.
2.2.1 Empirical studies of responsible ML practices. HCI scholarship examining the perceptions
and needs of responsible ML practitioners have identied key challenges [
48
,
99
], including limited
denitional consensus on key terms [
60
] and the underlying need to translate principles into
actionable guidance to catalyze transformative organizational change [
28
,
75
]. Practitioners often
work in multidisciplinary environments, where technical and non-technical stakeholders draw
on dierent epistemologies and perspectives [
86
], posing challenges to cohesive anticipation and
identication of harms and risks [
126
]. In terms of risk assessment specically, Raji et al. [
97
]
underscore how the often-rapid pace and piecemeal implementation of risk assessment inhibits
holistic forecasting of potential risks and their relationships to technical system components.
While there is a growing literature on practitioner needs, limited work has focused on identifying
existing social and ethical risk management practices and ML practitioners’ perspectives towards
safety engineering frameworks. Martelaro et al.’s [
78
] study of the applicability of hazard analysis
and the needs of practitioners is a notable exception. From an exploratory interview study with
eight participants, Martelaro et al. conclude existing hazard analysis tools from safety engineering
cannot readily support ML systems and highlight how lack of team incentives, the pace of industry
development, and underestimating the eort needed to create robust ML systems challenge im-
plementation of these tools. Nonetheless, Martelaro et al. emphasize frameworks are necessary to
support risk management for responsible ML practice.
2.3 Introducing safety engineering approaches to failure and hazard analysis
Safety engineering is a generic term for an assemblage of engineering analyses and management
practices designed to control dangerous situations arising in sociotechnical systems [
7
,
33
,
68
].
These analyses and practices identify potential hazards or system failures, understand their impact
on users or the public, investigate causes, develop appropriate controls to mitigate the potential
harms, and monitor systems [
113
]. Safety engineering crystallized as a discipline around WWII,
when military operators recognized losses and accidents were often the result of avoidable design
aws in technology and human factors [
122
]. Since then, implementation of safety engineering in
sociotechnical domains, such as medical devices and aerospace, has signicantly reduced accidents
and failures [103].
We motivate use of safety engineering for social and ethical risk management given its strength
in drawing attention to the relationships between risks, system design, and deployment [
30
,
97
].
As ML systems introduce interdependencies between the ML artifact, its operational environments,
and society at large [
101
], safety frameworks can provide a strong analytical grounding for risk
management [
33
]. Moreover, harms from ML systems are often recognized after they have oc-
curred [
98
] at which point mitigating them is signicantly more challenging and costly [
21
]. In this
study, we focus on two safety engineering techniques designed to identify and address undesired
outcomes early in development [
7
,
33
,
68
]: a failure analysis technique for improving reliability
(FMEA) and a hazard analysis technique for identifying unsafe system states (STPA).
, Vol. 1, No. 1, Article . Publication date: October 2023.
From plane crashes to algorithmic harm 5
2.3.1 Failure Mode and Eects Analysis (FMEA). FMEA, a long-standing reliability framework,
takes an analytic reduction (i.e. divide and conquer) approach to identifying and evaluating likeli-
hood of risk for potential failure modes (i.e. the mechanism of failure) for a technological system
or process [
21
]. FMEA has been used in high consequence projects, such as space shuttle [
54
]
and U.S. nuclear power plant safety [
73
]. The FMEA framework helps uncover potential failure
modes, identify the likelihood of risk, and address higher risk failure modes for a system (i.e.
bicycle), component (i.e. bicycle’s tire), or process (i.e. bicycle assembly) [
21
]. FMEA is a multi-step
framework, through which steps are iteratively performed by FMEA and system experts over the
development life cycle [21] (see also Fig 1):
(1)
List out the functions of a component/system OR steps of a process (e.g., everything the
system/process needs to perform).
(2)
Identify potential failure modes, or mechanisms by which each function or step can go wrong.
(3)
Identify the eect, or impact of a failure, and score its severity on a scale of 1 – 10 (least to
most severe).
(4)
Identify the cause, or why the failure mode occurs, and score its likelihood of occurrence on a
scale of 1 – 10 (least to most likely).
(5)
Identify controls, or how a failure mode could be detected, and score likelihood of detection on
a scale of 1 – 10 (least to most likely).
(6)
Calculate Risk Priority Number (RPN) by multiplying the three scores; higher RPN indicates
higher risk level.
(7) Develop recommended actions for each failure mode and prioritize based on RPN.
Fig. 1. Steps for conducting an FMEA [21]
2.3.2 System Theoretic Process Analysis (STPA). The hazard analysis method, STPA, is a relatively
new technique taking a system theoretic perspective towards safety [
68
]. It maps elements of a
system, their interactions, and examines potential hazards (i.e. sources of harm). While analytic
reduction requires a user of the tool to imagine interactions between components, modeling at the
system level is meant to capture emergent phenomena that are well-described only by component
interactions rather than individual component behavior. STPA has been employed in NASA’s space
program [52], the nuclear power industry [112], and the aviation industry [51].
In contrast to FMEA, the STPA process does not focus on reliability, failures, or risk likelihood.
Instead, STPA models the sociotechnical system, focusing on the structure between components
as well as control and feedback loops. Broadly, STPA encompasses the following steps, which are
meant to be iterative (across the model of a system) and cyclic (across a system’s lifecycle) (see Fig
2).
(1)
Dene the purpose of the analysis by identifying losses via outlining stakeholders, and their
values. System specic hazards and controls are then highlighted based on the specied loss.
, Vol. 1, No. 1, Article . Publication date: October 2023.
摘要：

Fromplanecrashestoalgorithmicharm:applicabilityofsafetyengineeringframeworksforresponsibleMLSHALALEHRISMANI,GoogleResearch,McGillUniversity,CanadaRENEESHELBY,Google,JusTechLab,AustralianNationalUniversity,U.S.AANDREWSMART,GoogleResearch,U.S.AEDGARJATHO,NavalPostgraduateSchool,U.S.AJOSHA.KROLL,NavalP...

展开>> 收起<<
From plane crashes to algorithmic harm applicability of safety engineering frameworks for responsible ML_2.pdf

共25页,预览5页

还剩页未读， 继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

相关推荐

更多
立即下载
分类：图书资源 价格：10玖币 属性：25 页 大小：750.24KB 格式：PDF 时间：2025-04-27

开通VIP享超值会员特权

  • 多端同步记录
  • 高速下载文档
  • 免费文档工具
  • 分享文档赚钱
  • 每日登录抽奖
  • 优质衍生服务

作者详情

相关内容

更多

热门标签

人际关系 配电装置 动力学连接体 力的合成 高考理综 全宋诗作者索引 公务员考试
/ 25
评分 收藏
分享
立即下载
客服
关注