1
The Extreme Cardiac MRI Analysis Challenge
under Respiratory Motion (CMRxMotion)
Shuo Wang1,2*, Chen Qin3†, Chengyan Wang4‡, Kang Wang1,2, Haoran Wang1,2, Chen Chen5, Cheng Ouyang5,
Xutong Kuang5, Chengliang Dai5, Yuanhan Mo6, Zhang Shi7, Chenchen Dai7, Xinrong Chen1,2, He Wang8, and
Wenjia Bai4
1Digitial Medical Research Center, Fudan Univeristy, China
2Shanghai Key Laboratory of MICCAI, China
3Department of Electrical and Electronic Engineering, Imperial College London, UK
4Human Phenome Institute, Fudan University, China
5Department of Computing, Imperial College London, UK
6School of Engineering, University of Oxford, UK
7Department of Radiology, Zhongshan Hospital Affiliated to Fudan University, China
8Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, China
(Summary of CMRxMotion Challenge Design)
Abstract—The quality of cardiac magnetic resonance (CMR)
imaging is susceptible to respiratory motion artifacts. The model
robustness of automated segmentation techniques in face of real-
world respiratory motion artifacts is unclear. This manuscript
describes the design of extreme cardiac MRI analysis challenge
under respiratory motion (CMRxMotion Challenge). The chal-
lenge aims to establish a public benchmark dataset to assess the
effects of respiratory motion on image quality and examine the
robustness of segmentation models. The challenge recruited 40
healthy volunteers to perform different breath-hold behaviors
during one imaging visit, obtaining paired cine imaging with
artifacts. Radiologists assessed the image quality and annotated
the level of respiratory motion artifacts. For those images with
diagnostic quality, radiologists further segmented the left ventri-
cle, left ventricle myocardium and right ventricle. The images
of training set (20 volunteers) along with the annotations are
released to the challenge participants, to develop an automated
image quality assessment model (Task 1) and an automated
segmentation model (Task 2). The images of validation set (5
volunteers) are released to the challenge participants but the
annotations are withheld for online evaluation of submitted
predictions. Both the images and annotations of the test set (15
volunteers) were withheld and only used for offline evaluation of
submitted containerized dockers. The image quality assessment
task is quantitatively evaluated by the Cohen’s kappa statistics
and the segmentation task is evaluated by the Dice scores and
Hausdorff distances.
Index Terms—Cardiac magnetic resonance, image quality
assessment, image segmentation, respiratory motion artifacts,
model robustness.
I. INTRODUCTION
Cardiac magnetic resonance (CMR) imaging is the cur-
rent gold-standard modality for evaluating cardiac structure
and function [1]. Machine learning-based approaches have
* Task 1 contact: shuowang@fudan.edu.cn
†Task 2 contact: c.qin15@imperial.ac.uk
‡CMR acquisition contact: wangcy@fudan.edu.cn
achieved remarkable performance in the CMR image seg-
mentation task [2], [3]. However, the model performance is
still challenged by inconsistent imaging environments (e.g.,
vendors and protocols) [4], population shifts (normal v.s.
pathological cases) [5] and unexpected human behaviors (e.g.,
body movements) in clinical practice. It is useful to investigate
potential failure modes [6] by exposing a trained segmentation
model to extreme cases in a ‘stress test’. To date, existing
challenges focus on the vendor variability [4] and anatomical
structure variations [5] while the implications of human be-
haviors are less explored. For CMR image analysis, respiration
motion is one of the major problems [7]. Patients may not
be able to follow the breath-hold instructions well, which is
common in heart failure cases or children. The poor breath-
hold behaviors result in degraded image quality and inaccurate
analysis [8]. Therefore, we launch the extreme cardiac MRI
analysis challenge under respiratory motion (CMRxMotion
Challenge) 1, to establish a public benchmark dataset to assess
the effects of respiratory motion on CMR imaging quality and
examine the robustness [9] of automated segmentation models.
To curate such an extreme CMR dataset with respiratory
motion artifacts, one retrospective way is to screen the images
stored in hospital database and identify those problematic ones.
But this requires considerable human efforts and may also
bring about confounding factors such as vendors, scan protocol
and pathology. Instead, we design a prospective study that
health volunteers are recruited to perform different breath-
hold behaviors during one imaging visit. As the confounding
factors of MRI equipment and scan protocols are controlled,
this extreme CMR dataset is established in specific to respi-
ratory motion artifacts. The rest of the manuscript provides
a summary of CMRxMotion challenge design, including the
data acquisition, annotation, tasks, evaluation metric, ranking
1http://cmr.miccai.cloud
arXiv:2210.06385v1 [eess.IV] 12 Oct 2022