Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Experts Santosh T.Y.S.S1and Shanshan Xu1and Oana Ichim2and Matthias Grabmair1

2025-05-06 0 0 1.09MB 19 页 10玖币

侵权投诉

Deconfounding Legal Judgment Prediction for European Court of Human

Rights Cases Towards Better Alignment with Experts

Santosh T.Y.S.S1∗and Shanshan Xu1∗and Oana Ichim2and Matthias Grabmair1

1School of Computation, Information, and Technology; Technical University of Munich, Germany

2Graduate Institute of International and Development Studies, Geneva, Switzerland

{santosh.tokala, shanshan.xu, matthias.grabmair}@tum.de

oana.ichim@graduateinstitute.ch

Abstract

This work demonstrates that Legal Judgement

Prediction systems without expert-informed

adjustments can be vulnerable to shallow, dis-

tracting surface signals that arise from cor-

pus construction, case distribution, and con-

founding factors. To mitigate this, we use do-

main expertise to strategically identify statis-

tically predictive but legally irrelevant infor-

mation. We adopt adversarial training to pre-

vent the system from relying on it. We eval-

uate our deconfounded models by employing

interpretability techniques and comparing to

expert annotations. Quantitative experiments

and qualitative analysis show that our decon-

founded model consistently aligns better with

expert rationales than baselines trained for pre-

diction only. We further contribute a set of

reference expert annotations to the validation

and testing partitions of an existing benchmark

dataset of European Court of Human Rights

cases.

1 Introduction

The task of Legal Judgment Prediction (LJP) has

recently gained increasing attention in the legal

and mainstream NLP communities (Aletras et al.,

2016;Zhong et al.,2018;Medvedeva et al.,2020;

Liu et al.,2019;Sert et al.,2021). Legal cases

are resolved through the exchange of arguments

in front of a decision body by lawyers who rep-

resent litigating parties. This typically involves

evidential reasoning, the determination of relevant

rules from sources of law (e.g., codes, regulations,

precedent), their application to the case, and the bal-

ancing of legal and societal values. In the NLP con-

text, LJP takes the form of classifying the outcome

of a case from some textual representation of its

speciﬁc facts, effectively skipping legal reasoning.

This forms a counterpoint to knowledge-focused

*These authors contributed equally to this work

Spurious Tokens

ECtHR

Decision Fact

Statement

Case

Outcome

Decision Tree

Classiﬁers Spurious Attributes

Respondent State

Doc. Length

Legal

Expert

Review

Hierarchical BERT

Classiﬁer

Adversarial

Decounfounder

Evaluation

IG Focus Map

Expert Rationale

Figure 1: Our deconfounding experiment architecture

approaches to outcome prediction (e.g., Brüning-

haus and Ashley,2005;Branting,2013;Grabmair,

2017) that connect to a lawyer’s understanding of

the domain but also require substantial knowledge

engineering.

This carries particular risk in the legal domain,

where systems may rely on data elements that are

statistically predictive but legally irrelevant, or even

forbidden as decision criteria (e.g., the race of an

accused person). This can lead to undesirable con-

sequences, ranging from suboptimal litigation strat-

egy decisions, ﬂawed inference about factors pre-

dictive for the outcome, to disparate impact of de-

cisions across groups that are to be treated equally.

If legal decisions are to be informed by predictive

systems processing textual case descriptions, then

such systems must strive to be as closely aligned

with legally relevant and permissible parts of the

input as possible.

In this work, we focus on LJP for the European

Court of Human Rights (ECtHR), which adjudi-

cates complaints by individuals against states about

alleged violations of their rights as enshrined in

the European Convention of Human Rights. We

trained deep neural models on four tasks across

two existing, related datasets (Chalkidis et al.,2019,

2022a) around predicting such violations alleged by

the claimant and decided by the court. We ﬁnd that

arXiv:2210.13836v1 [cs.CL] 25 Oct 2022

the models substantially base their predictions on

aspects of the text that correlate with the outcome

but either have no legal bearing or are forbidden

nationality-related information that stem from the

distribution of cases arising at the court.

To improve the alignment of model focus with

legal expert understanding, we apply a series of

deconfounding measures, including a vocabulary-

based method which identiﬁes predictive tokens

using a simple model. The third author, who is an

ECtHR expert, then identiﬁes distractors among

them. The distracting signal can subsequently be

removed from the encodings via adversarial train-

ing. This procedure is an effective way of engag-

ing with domain experts and obtaining information

about what the model should be steered away from

by means of deconfounding, rather than trying to

attract the model towards relevant elements via ex-

pensive data collection for supervised training. For

simplicity, throughout this paper, we use ‘decon-

founding’ in an inclusive sense as the mitigation of

distracting effects of (a) confounders in the statis-

tical sense that inﬂuence both the dependent and

independent variables, (b) reverse causation rela-

tionships, and (c) other attributes that spuriously

correlate with the target variable. See Fig. 1for an

overview of our experiment design.

We evaluate our trained and deconfounded mod-

els with regard to an alignment of its explana-

tion rationales with (1) a dataset of expert pas-

sage relevance assessments we collected and will

make available to community as a supplement to

Chalkidis et al. (2019), and (2) on expert relevance

assessments published as part of Chalkidis et al.

(2021). Our results show that our deconfounding

steps succeed in improving the model focus align-

ment with expert-identiﬁed, relevant patterns on

both sets of reference annotations.

In sum, we make the following contributions:

•

We introduce an expert-informed deconfound-

ing method which identiﬁes distracting ef-

fects from confounders and spurious corre-

lations using a simple model, and mitigates

them through adversarial training, thus help-

ing to improve the alignment of the model

focus with legal expert rationales.

•

We empirically evaluate this method on four

tasks in legal judgment prediction on ECtHR

data and show that our model consistently

aligns better with expert rationales than a base-

line trained for the prediction target only.

•

We release a set of gold rationales annotated

by an ECtHR expert as a supplement to an

existing dataset to facilitate future work on

deriving more useful insight from trained pre-

dictive systems in the legal domain.*

2 Related Work

LJP as an NLP task has been tackled using n-

gram representations (e.g., Aletras et al.,2016;

Medvedeva et al.,2020), word embeddings and

domain models (Branting et al.,2021), and deep

neural networks (e.g., Chalkidis et al.,2019;Ma

et al.,2021;Xu et al.,2020). Special attention must

be given to the origin of the text from which the

prediction is to be made. Medvedeva et al. (2021,

2022) recharacterize LJP on texts produced before

the outcome is known as ‘forecasting’ and observes

that most current works ‘classify’ judgments based

on the data compiled after the outcome has been

determined. They also ﬁnd that forecasting is a

harder task. This result is consistent with our ﬁnd-

ing of confounding effects from text production

by the ECtHR, resulting in a prediction from fact

descriptions that were inﬂuenced by the decision.

Moverover, the relationship between the infor-

mation LJP models rely on and legal expert anal-

ysis of texts remains underexplored. Bhambhoria

et al. (2021) ﬁnd that transformer-based models ex-

ploit spurious correlations and that simple models,

such as XGBoost, can achieve similar performance.

Chalkidis et al. (2021) extract model rationales for

alleged violation prediction and observes limited

overlap with expert markup. Similarly, a small

study in Branting et al. (2021) ﬁnds that users do

not perceive case prediction-derived highlighting

as useful in making predictions themselves. Our

work contributes to this state of the art by using

adversarial deconfounding to improve the overlap

between what systems predict from with what legal

experts consider relevant.

Deconfounding

A growing number of works have

raised awareness that deep neural models may ex-

ploit spurious statistical patterns and take erroneous

shortcuts (McCoy et al.,2019;Bender and Koller,

2020;Geirhos et al.,2020). A common method of

mitigating this is adversarial learning. Pryzant et al.

2018 use a gradient reversal layer (Ganin et al.,

2016) to deconfound lexicons in text classiﬁcation.

Our rationales and code are available at

https://github.com/TUMLegalTech/deconfounding_echr_

emnlp22

Other domains that adopt adversarial training to

eliminate confounders include bioinformatics (Din-

cer et al.,2020) and political science (Roberts et al.,

2020). Many existing works on identifying short-

cuts focus on situations where these patterns are

known in advance and may require potentially ex-

pensive data collection. In fairness-focused legal

NLP, Chalkidis et al. (2022b) observe and remedy

group disparities in LJP performance on the EC-

tHR informed by metadata attributes (respondent

state, applicant gender, applicant age). We extend

this to explainability in LJP by involving a legal

expert in a procedure that allows an efﬁcient, incre-

mental identiﬁcation of distracting information, as

well as its removal via adversarial training.

Interpretability

We employ interpretability tech-

niques to evaluate model alignment with expert

rationales. Danilevsky et al. (2020) reviews and

categorizes the main current interpretability meth-

ods. Though initial works (Ghaeini et al.,2018;

Lee et al.,2017) used attention scores as explana-

tion for model decisions, Bastings and Filippova

(2020); Serrano and Smith (2019) point out that

saliency methods, such as gradient based methods

(Sundararajan et al.,2017;Li et al.,2016), propaga-

tion based methods (Bach et al.,2015), occlusion

based methods (Zeiler and Fergus,2014), and sur-

rogate model based methods (Ribeiro et al.,2016)

are better suited for explainability analysis. How-

ever, the reliability and informativeness of these

methods remains an open research problem. Our

model uses the currently most commonly used In-

tegrated Gradients (IG) (Sundararajan et al.,2017),

which computes the gradient of the model’s output

with respect to its input features.

3 ECtHR Tasks & Datasets

The ECtHR has been the subject of substantial prior

work in LJP. We use two datasets for model training

and evaluation: First, for

binary violation

we use

the dataset by Chalkidis et al. (2019) of approx. 11k

case fact statements, where the target is to predict

whether the court has found at least one convention

article to be violated. To evaluate alignment, we

annotate 50 (25 each) expert rationales for cases

from both the development and test partitions (See

App. Cfor the annotation process). Second, for

article-speciﬁc violation

, we use the LexGLUE

dataset by Chalkidis et al. (2022a), which consists

of 11k case fact statements along with information

about which convention articles have been alleged

to be violated, and which the court has found to

be violated. For alignment, we merge this data

with the 50 test set rationales from Chalkidis et al.

(2021). While both datasets stem from the EC-

tHR’s public database, they differ in case facts and

outcome distribution as we explain in Sec. 3.1. The

input texts consist of each case’s FACTS section

extracted from ECtHR judgments. This section is

drafted by court staff over the course of the case

proceedings. While it does not contain the out-

come explicitly, it is not ﬁnalized before the ﬁnal

decision has been determined, potentially creating

confounding effects.

We conduct experiments on four LJP tasks:

Task J - Binary Violation

For our task

, the

model is given a fact statement and is asked to

predict whether or not any article of the conven-

tion has been violated. We train our models on

Chalkidis et al. (2019) and evaluate alignment on

the set of expert rationales we collected.

Task B - Article Allegation

We train and evaluate

on LexGLUE’s ECtHR B,

where the fact descrip-

tion is the basis to predict the set of convention arti-

cles that the claimant alleges to have been violated.

It can be conceptualized as topic classiﬁcation in

that the system needs to identify suitable candidate

articles (e.g., the right to respect for private and

family life) from fact statements (e.g., about gov-

ernment surveillance). We test alignment on the

expert rationales by Chalkidis et al. (2021).

Task A - Article Violation

We also experiment

with LexGLUE’s ECtHR A, which is to predict

which of the convention’s articles has been deemed

violated by the court from a case’s fact descrip-

tion. Task A is a more difﬁcult version of task B,

where both an identiﬁcation of suitable articles and

a prediction of their violation must be performed.

For alignment, we again use the expert rationales

by Chalkidis et al. (2021), which are technically

intended for task ECtHR B, but which we consider

to also be suitable for an evaluation of task A.*

Task A|B - Article Violation given Allegation

We further disentangle the LexGLUE tasks and

pose ECtHR A|B. Given the facts of a case and

the allegedly violated articles, the model should

The LexGLUE dataset does not contain metadata (case id,

Respondent state etc); in this work we use an enriched version

of the same dataset by Mathurin Aché.

The annotation explanations in (Chalkidis et al.,2021)

state that “The annotator selects the factual paragraphs that

“clearly” indicate allegations for the selected article(s)”. We

hypothesize that the so annotated passages contain information

that is legally relevant for the violation as well.

predict which (if any) speciﬁc articles have been

violated. This task reﬂects the legal process, as

the court is aware of allegations made by the appli-

cants when deciding. Providing information about

the allegations shifts the nature of the task from

topic classiﬁcation to article-speciﬁc violation/non-

violation prediction, thus refocusing the model and

ideally leading to violation-speciﬁc explanations.

3.1 Data Distribution & Preprocessing

In order to facilitate model alignment, we worked

with our ECtHR expert to identify shallow predic-

tion signals in the fact statements that are unrelated

to the legal merits of the complaint.

3.1.1 Length and Respondent State

For the task J dataset of Chalkidis et al. 2019,

we ﬁnd that the distribution of fact description

length (number of sentences) and the distribution

of respondent states are different between the two

classes (see Appendix A). We hence account for the

identity of the respondent state and the length of the

fact descriptions via our deconfounding procedure

for both datasets.

3.1.2 Accounting for Inadmissible Cases

We also observe in the task J dataset that the mag-

nitudes of the running paragraph numbers differ

between the classes, and that the single word “rep-

resented” strongly correlates with the positive class.

This phenomenon arises because 2.6k of the 7k

training cases are ‘inadmissible’ cases labeled as

‘non-violation’. Legally, inadmissible cases are not

necessarily ‘non-violation’ as inadmissibility re-

lates to complaints not fulﬁlling the court’s formal

or procedural criteria.

In such cases, the court

does not examine the merits of the application. The

more interesting non-violation cases are such that

are admissible, but in which no violation of the

convention has been found. The single negative

class contains instances of both inadmissible and

admissible-but-no-violation-found cases. As ex-

plained above, the input texts of Chalkidis et al.

2019 are extracted from the FACTS section of full

ECtHR decisions. In inadmissible cases, the ap-

plicant’s background information can typically be

found at the beginning of that section. We found

For example, the applicants lodge the complaint outside

the time limit after the ﬁnal domestic judicial decision or fail

to exhaust required domestic remedies before complaining to

the ECtHR, etc. It should be noted that the majority of inad-

missible cases are decided by single judges and not available

on the public database HUDOC.

that almost all inadmissible case facts start with

the same formulaic sentence stating the applicant’s

name, nationality, and legal representation. This

speciﬁc sentence is absent from the texts of ad-

missible cases (violation and non-violation), where

that information is part of a separate PROCEDURE

section not included in the dataset. Moreover, due

to the PROCEDURE section preceding the FACTS

section in admissible cases, the running paragraph

numbers appearing in FACTS sections of inadmis-

sible cases are smaller than those of the admissible

cases. If not remedied, these phenomena provide

a considerable predictive signal for the label and

distract the system from legally relevant informa-

tion. In our experiments, we hence remove para-

graph numbers from the input via preprocessing

and account for distractor vocabulary via our de-

confounding procedure described in Sec. 4. Still,

the nature of task J remains unchanged and requires

the system to classify the outcomes of a collection

of both admissible and inadmissible cases.

3.1.3 Article-Speciﬁc Violation

By contrast, the more recent LexGLUE dataset only

contains admissible cases and corresponding infor-

mation about which articles the claimant has al-

leged to have been violated (for task B) along with

those that the court has found to have been violated,

if any (task A). The collection covers 10 different

convention articles that make up the largest share

of ECtHR jurisprudence. Each article has been

alleged in a partition of the cases, and has been

found to be violated in a subset of these.

For a

given article in task B, all cases in which it has been

alleged can be considered positive instances while

the remaining cases are negatives. We consider task

B as akin to topic classiﬁcation, where the rights

enshrined in the convention articles (e.g., Art. 6:

right to a fair trial; Art. 1 Protocol 1: protection of

property, etc.) may correlate with certain case fact

language (e.g., related to law enforcement or ex-

propriation, respectively). Task A incorporates this

step and adds violation prediction per article, which

is more difﬁcult in principle. However, we observe

that a few articles account for a large portion of the

data and the conditional probability of a positive

violation label in task A given its allegation labels

from task B can be very high (see App. B). This

A few cases exist where the court refocuses the issues and

ﬁnds a violation of an article that has not been alleged, but in

the dataset they only occur in a negligibly small number of

instances. (see App. Sec. B)

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

DeconfoundingLegalJudgmentPredictionforEuropeanCourtofHumanRightsCasesTowardsBetterAlignmentwithExpertsSantoshT.Y.S.S1andShanshanXu1andOanaIchim2andMatthiasGrabmair11SchoolofComputation,Information,andTechnology;TechnicalUniversityofMunich,Germany2GraduateInstituteofInternationalandDevelopmentStud...

展开>> 收起<<

Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Experts Santosh T.Y.S.S1and Shanshan Xu1and Oana Ichim2and Matthias Grabmair1.pdf

共19页,预览4页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Experts Santosh T.Y.S.S1and Shanshan Xu1and Oana Ichim2and Matthias Grabmair1

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: