From plane crashes to algorithmic harm applicability of safety engineering frameworks for responsible ML_2

2025-04-27 0 0 750.24KB 25 页 10玖币

侵权投诉

From plane crashes to algorithmic harm: applicability of

safety engineering frameworks for responsible ML

SHALALEH RISMANI, Google Research, McGill University, Canada

RENEE SHELBY, Google, JusTech Lab, Australian National University, U.S.A

ANDREW SMART, Google Research, U.S.A

EDGAR JATHO, Naval Postgraduate School, U.S.A

JOSH A. KROLL, Naval Postgraduate School, U.S.A

AJUNG MOON, McGill University, Canada

NEGAR ROSTAMZADEH, Google Research, Canada

Inappropriate design and deployment of machine learning (ML) systems leads to negative downstream social

and ethical impact – described here as social and ethical risks – for users, society and the environment. Despite

the growing need to regulate ML systems, current processes for assessing and mitigating risks are disjointed

and inconsistent. We interviewed 30 industry practitioners on their current social and ethical risk management

practices, and collected their rst reactions on adapting safety engineering frameworks into their practice

– namely, System Theoretic Process Analysis (STPA) and Failure Mode and Eects Analysis (FMEA). Our

ndings suggest STPA/FMEA can provide appropriate structure toward social and ethical risk assessment

and mitigation processes. However, we also nd nontrivial challenges in integrating such frameworks in

the fast-paced culture of the ML industry. We call on the ML research community to strengthen existing

frameworks and assess their ecacy, ensuring that ML systems are safer for all people.

CCS Concepts:

•Social and professional topics →Computing / technology policy

;

•General and

reference →Evaluation;Surveys and overviews.

Additional Key Words and Phrases: empirical study, safety engineering, machine learning, social and ethical

risk

ACM Reference Format:

Shalaleh Rismani, Renee Shelby, Andrew Smart, Edgar Jatho, Josh A. Kroll, AJung Moon, and Negar Ros-

tamzadeh. 2023. From plane crashes to algorithmic harm: applicability of safety engineering frameworks for

responsible ML. 1, 1 (October 2023), 25 pages. https://doi.org/XXXXXXX.XXXXXXX

1 INTRODUCTION

During a panel at the 1994 ACM Conference on Human Factors in Computing Systems (CHI),

prominent scholars from dierent disciplines convened to discuss "what makes a good computer

system good." Panelists highlighted considerations for safety, ethics, user perspectives, and societal

structures as critical elements for making a good system [

]. Almost 28 years later, we posit that

Authors’ addresses: Shalaleh Rismani, Google Research, McGill University, Montreal, Canada; Renee Shelby, Google, JusTech

Lab, Australian National University, San Francisco, U.S.A; Andrew Smart, Google Research, San Francisco, U.S.A; Edgar

Jatho, Naval Postgraduate School, Monterey, U.S.A; Josh A. Kroll, Naval Postgraduate School, Monterey, U.S.A; AJung Moon,

McGill University, Montreal, Canada; Negar Rostamzadeh, Google Research, Montreal, Canada.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee

provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and

the full citation on the rst page. Copyrights for components of this work owned by others than ACM must be honored.

Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires

prior specic permission and/or a fee. Request permissions from permissions@acm.org.

XXXX-XXXX/2023/10-ART $15.00

https://doi.org/XXXXXXX.XXXXXXX

, Vol. 1, No. 1, Article . Publication date: October 2023.

arXiv:2210.03535v1 [cs.HC] 6 Oct 2022

2 Rismani et al.

these epistemological perspectives need to be in deeper conversation for designing and assessing

machine learning (ML) systems that challenge conventional understanding of safety and harm.

The development and use of ML systems can adversely impact people, communities, and society

at large [

118

125

], including inequitable resource allocation [

106

], perpetuating

normative narratives about people and social groups [

119

], and the entrenchment of social

inequalities [

]. We frame these adverse impacts broadly as social and ethical risks. To manage

such risks, quantitative [

], qualitative [

], and epistemological frameworks [

] have been proposed. In parallel, active regulatory and standards activities are taking place

internationally [

–

]. Despite the rapidly evolving discourse, there is limited empirical

understanding of how proposed social and ethical risk management tools have been adopted by

practitioners [36, 74].

Inspired by the 1994 panelists, we examine the dialogue between safety engineering frameworks

and understandings of social and ethical risks of

systems. First, we report on ethical and

social risk management practices currently used in the industry. Second, we take a developmental

approach to examine how safety engineering frameworks can improve existing practices. We

chose two of the most successful safety engineering frameworks used in other sociotechnical

domains [

117

]: Failure Mode and Eect Analysis (FMEA) [

] and System Theoretic Process

Analysis (STPA) [67, 91], which we describe in detail in Section 2.

We conducted 30 semi-structured in-depth interviews with industry practitioners who shared

their current practices used to assess and mitigate social and ethical risks. We introduced the two

safety engineering frameworks, inviting them to envision how they might employ them to assess

ethical and social risk of ML systems. The results of our study address the following research

questions:

•RQ1: Which practices do ML practitioners use to manage social and ethical risks today?

•RQ2

: What challenges do practitioners face in their attempts to manage social and ethical

risks?

•RQ3

: How could safety engineering frameworks such as FMEA and STPA inform and improve

current practices? What advantages and disadvantages of each method do ML practitioners

identify?

We contribute to the emerging research on managing social and ethical risk of

systems in

human-computing scholarship and responsible ML communities by oering:

•An overview of how practitioners dene, assess and mitigate social and ethical risks;

•An analysis of the corresponding challenges when implementing these practices;

•

A set of insights on how FMEA and STPA could inform existing practices along with their

perceived advantages and disadvantages;

•Future research directions and calls to action for HCI and responsible ML scholars.

Our ndings illustrate safety engineering frameworks provide valuable structure for investigating

how social and ethical risks emerge from ML systems design and integration in a given context.

However, successful adaptation of these frameworks requires solutions to existing organizational

challenges for operationalizing formal risk management practices. Moreover, results of our work

motivate further theoretical and applied research on adaptation of such frameworks. The remainder

of this paper is organized as follows. We start by providing an overview of current discourse in

responsible ML development and contextualize the relevance of the safety engineering frameworks

(Section 2). We outline our interview protocol and analysis methods in Section 3, followed by

highlighting key ndings in Section 4. We discuss the value and shortcomings of applying safety

engineering frameworks in light of current practices and call on the research community to further

, Vol. 1, No. 1, Article . Publication date: October 2023.

From plane crashes to algorithmic harm 3

examine and strengthen these frameworks for ethical and social risk management of

systems

in Section 5.

2 BACKGROUND

Analyzing social and ethical implications of algorithmic systems is not new to computing researchers

and practitioners [

]. In the literature, terms such as harm [

118

], failure [

], and

risk [

125

] are often used to describe adverse impacts of ML systems. While there is currently no

agreed upon denition of these terms and their relationships, we use the phrase social and ethical

risk to frame broadly the adverse social and ethical implications ML systems can have on users,

society, and the environment. This working denition provides conceptual consistency in this

paper, and is not meant to be normative. In the remainder of this section, we contextualize current

discourses on social and ethical risks in ML to situate our study design, ndings, and discussion.

We highlight current epistemological perspectives and tools for responsible

development, and

detail the safety engineering frameworks (FMEA and STPA).

2.1 Epistemological perspectives for anticipating and mitigating harms of ML systems

Scholars have proposed various methods for anticipating social and ethical impacts [

109

Anticipating harm involves thinking about the values [

111

] and aordances of ML systems [

with specic attention to how social norms and power dynamics constitutively shape adverse

impacts of ML systems [

]. The process of anticipation is aided by critical epistemologies that

center the needs and standpoints of socially oppressed groups, including critical race theory [

46, 58, 89], post-colonial theories [82], and queer [114] and feminist HCI [9].

As social and ethical impacts are co-constituted through the interplay of technical system com-

ponents and the social world [

], design methodologies attentive to these dynamics support more

meaningful harms anticipation and mitigation. For instance, Value Sensitive Design that examines

what value tensions ML systems create or resolve [

124

], supports increased stakeholder coordi-

nation [

121

] and consideration of technology from dierent social standpoints and perspectives

[

]. Similarly, participatory design methods can center the needs of users, communities, and other

stakeholders often excluded from the design process [

130

] or algorithmic governance [

especially when incorporating feminist epistemologies [

]. Speculative design can also help

designers imagine more socially just and racially equitable technological futures [

]. While these

epistemological perspectives and frameworks do not explicitly assess risk, they provide theoretical

grounds for examining and mitigating social and ethical risk.

2.2 Responsible ML tools, processes, and emerging regulations

With increased deployment of ML systems and reported harms [

125

], there is movement

towards formalizing quantitative and qualitative tools for responsible

development. Tradi-

tionally,

system evaluations [

104

] prioritized assessing and optimizing for a narrow set

of performance metrics, mistakenly treating these measurements (e.g., accuracy of a test set)

as a target rather than proxy for certain risks [

]. Recognizing these shortcomings [

scholars proposed alternative methods to enable more comprehensive evaluation. These methods

include assessing computational fairness with alternative statistical denitions [

quantifying model interpretability based on statistical properties [

], evaluating robustness

to distribution shift [

116

] and examining model performance when exposed to adversarial

examples [34, 105, 128, 129].

In parallel, signicant eort has also focused on developing mixed-method (qualitative and

quantitative) processes to increase accountability and assess ML systems contextually. Scholars

have proposed model cards [

], datasheets [

] and auditing tools [

110

] to improve the

, Vol. 1, No. 1, Article . Publication date: October 2023.

4 Rismani et al.

transparency and quality of model and data practices. Human right and algorithmic impact assess-

ments aid identication of potential societal level harms by examining model deployment in a given

context [

]. Similarly, scholars have developed contextual methods of assessing fairness by

focusing attention on the situated power dynamics of where systems are deployed [

107

127

] and

transparency [

123

] of

systems. Parallel to technique development, there is a rapidly emerging

set of international standards [

], policies [

120

], and regulatory frameworks [

] that examine ML

systems from a risk-based perspective.

2.2.1 Empirical studies of responsible ML practices. HCI scholarship examining the perceptions

and needs of responsible ML practitioners have identied key challenges [

], including limited

denitional consensus on key terms [

] and the underlying need to translate principles into

actionable guidance to catalyze transformative organizational change [

]. Practitioners often

work in multidisciplinary environments, where technical and non-technical stakeholders draw

on dierent epistemologies and perspectives [

], posing challenges to cohesive anticipation and

identication of harms and risks [

126

]. In terms of risk assessment specically, Raji et al. [

]

underscore how the often-rapid pace and piecemeal implementation of risk assessment inhibits

holistic forecasting of potential risks and their relationships to technical system components.

While there is a growing literature on practitioner needs, limited work has focused on identifying

existing social and ethical risk management practices and ML practitioners’ perspectives towards

safety engineering frameworks. Martelaro et al.’s [

] study of the applicability of hazard analysis

and the needs of practitioners is a notable exception. From an exploratory interview study with

eight participants, Martelaro et al. conclude existing hazard analysis tools from safety engineering

cannot readily support ML systems and highlight how lack of team incentives, the pace of industry

development, and underestimating the eort needed to create robust ML systems challenge im-

plementation of these tools. Nonetheless, Martelaro et al. emphasize frameworks are necessary to

support risk management for responsible ML practice.

2.3 Introducing safety engineering approaches to failure and hazard analysis

Safety engineering is a generic term for an assemblage of engineering analyses and management

practices designed to control dangerous situations arising in sociotechnical systems [

These analyses and practices identify potential hazards or system failures, understand their impact

on users or the public, investigate causes, develop appropriate controls to mitigate the potential

harms, and monitor systems [

113

]. Safety engineering crystallized as a discipline around WWII,

when military operators recognized losses and accidents were often the result of avoidable design

aws in technology and human factors [

122

]. Since then, implementation of safety engineering in

sociotechnical domains, such as medical devices and aerospace, has signicantly reduced accidents

and failures [103].

We motivate use of safety engineering for social and ethical risk management given its strength

in drawing attention to the relationships between risks, system design, and deployment [

As ML systems introduce interdependencies between the ML artifact, its operational environments,

and society at large [

101

], safety frameworks can provide a strong analytical grounding for risk

management [

]. Moreover, harms from ML systems are often recognized after they have oc-

curred [

] at which point mitigating them is signicantly more challenging and costly [

]. In this

study, we focus on two safety engineering techniques designed to identify and address undesired

outcomes early in development [

]: a failure analysis technique for improving reliability

(FMEA) and a hazard analysis technique for identifying unsafe system states (STPA).

, Vol. 1, No. 1, Article . Publication date: October 2023.

From plane crashes to algorithmic harm 5

2.3.1 Failure Mode and Eects Analysis (FMEA). FMEA, a long-standing reliability framework,

takes an analytic reduction (i.e. divide and conquer) approach to identifying and evaluating likeli-

hood of risk for potential failure modes (i.e. the mechanism of failure) for a technological system

or process [

]. FMEA has been used in high consequence projects, such as space shuttle [

]

and U.S. nuclear power plant safety [

]. The FMEA framework helps uncover potential failure

modes, identify the likelihood of risk, and address higher risk failure modes for a system (i.e.

bicycle), component (i.e. bicycle’s tire), or process (i.e. bicycle assembly) [

]. FMEA is a multi-step

framework, through which steps are iteratively performed by FMEA and system experts over the

development life cycle [21] (see also Fig 1):

(1)

List out the functions of a component/system OR steps of a process (e.g., everything the

system/process needs to perform).

(2)

Identify potential failure modes, or mechanisms by which each function or step can go wrong.

(3)

Identify the eect, or impact of a failure, and score its severity on a scale of 1 – 10 (least to

most severe).

(4)

Identify the cause, or why the failure mode occurs, and score its likelihood of occurrence on a

scale of 1 – 10 (least to most likely).

(5)

Identify controls, or how a failure mode could be detected, and score likelihood of detection on

a scale of 1 – 10 (least to most likely).

(6)

Calculate Risk Priority Number (RPN) by multiplying the three scores; higher RPN indicates

higher risk level.

(7) Develop recommended actions for each failure mode and prioritize based on RPN.

Fig. 1. Steps for conducting an FMEA [21]

2.3.2 System Theoretic Process Analysis (STPA). The hazard analysis method, STPA, is a relatively

new technique taking a system theoretic perspective towards safety [

]. It maps elements of a

system, their interactions, and examines potential hazards (i.e. sources of harm). While analytic

reduction requires a user of the tool to imagine interactions between components, modeling at the

system level is meant to capture emergent phenomena that are well-described only by component

interactions rather than individual component behavior. STPA has been employed in NASA’s space

program [52], the nuclear power industry [112], and the aviation industry [51].

In contrast to FMEA, the STPA process does not focus on reliability, failures, or risk likelihood.

Instead, STPA models the sociotechnical system, focusing on the structure between components

as well as control and feedback loops. Broadly, STPA encompasses the following steps, which are

meant to be iterative (across the model of a system) and cyclic (across a system’s lifecycle) (see Fig

2).

(1)

Dene the purpose of the analysis by identifying losses via outlining stakeholders, and their

values. System specic hazards and controls are then highlighted based on the specied loss.

, Vol. 1, No. 1, Article . Publication date: October 2023.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

Fromplanecrashestoalgorithmicharm:applicabilityofsafetyengineeringframeworksforresponsibleMLSHALALEHRISMANI,GoogleResearch,McGillUniversity,CanadaRENEESHELBY,Google,JusTechLab,AustralianNationalUniversity,U.S.AANDREWSMART,GoogleResearch,U.S.AEDGARJATHO,NavalPostgraduateSchool,U.S.AJOSHA.KROLL,NavalP...

展开>> 收起<<

From plane crashes to algorithmic harm applicability of safety engineering frameworks for responsible ML_2.pdf

共25页,预览5页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

From plane crashes to algorithmic harm applicability of safety engineering frameworks for responsible ML_2

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: