Smart-Badge A wearable badge with multi-modal sensors for kitchen activity recognition

2025-05-03 0 0 8.96MB 11 页 10玖币

侵权投诉

Smart-Badge: A wearable badge with multi-modal sensors for

kitchen activity recognition

MENGXI LIU, SUNGHO SUH, BO ZHOU, AGNES GRÜNERBL, and PAUL LUKOWICZ,

Ger-

man Research Center for Articial Intelligence (DFKI), Germany

Human health is closely associated with their daily behavior and environment. However, keeping a healthy lifestyle is still

challenging for most people as it is dicult to recognize their living behaviors and identify their surrounding situations to

take appropriate action. Human activity recognition is a promising approach to building a behavior model of users, by which

users can get feedback about their habits and be encouraged to develop a healthier lifestyle. In this paper, we present a smart

light wearable badge with six kinds of sensors, including an infrared array sensor MLX90640 oering privacy-preserving,

low-cost, and non-invasive features, to recognize daily activities in a realistic unmodied kitchen environment. A multi-

channel convolutional neural network (MC-CNN) based on data and feature fusion methods is applied to classify 14 human

activities associated with potentially unhealthy habits. Meanwhile, we evaluate the impact of the infrared array sensor on the

recognition accuracy of these activities. We demonstrate the performance of the proposed work to detect the 14 activities

performed by ten volunteers with an average accuracy of 92.44 % and an F1 score of 88.27 %.

CCS Concepts: •Human-centered computing →Ubiquitous computing.

Additional Key Words and Phrases: Multi-sensor Wearable Device, Kitchen Activity Recognition, Sensor Fusion

ACM Reference Format:

Mengxi Liu, Sungho Suh, Bo Zhou, Agnes Grünerbl, and Paul Lukowicz. 2022. Smart-Badge: A wearable badge with multi-

modal sensors for kitchen activity recognition. In Proceedings of the 2022 ACM International Joint Conference on Pervasive and

Ubiquitous Computing (UbiComp/ISWC ’22 Adjunct), September 11–15, 2022, Cambridge, United Kingdom. ACM, New York, NY,

USA, 11 pages. https://doi.org/10.1145/3544793.3560391

1 INTRODUCTION

Human activity and the surrounding situation in daily life have a substantial eect on human health and life

quality. Although keeping a healthy lifestyle has emerged as a popular topic in the crowd, it is still dicult

for many people, especially old people, to recognize their behavior and identify situations around them daily.

Human activity recognition has been a promising tool for both users understanding their behavior and doctors

diagnosing potential diseases, by which life quality of humans could be increased signicantly. Therefore,

human activity recognition is an important research direction in pervasive computing [

] and has been widely

investigated over the decades. With the rapid development of sensor technology and articial intelligence

algorithms, various solutions for human activity recognition based on novel sensor modality [

] and

machine learning algorithms [

] have been proposed to extract comprehensive context from human

activities predominantly, including body position-related, body action-related and body status-related context [

]

with the use of wearable devices or ambient sensors, by which user’s lifestyle can be evaluated, and a behavior

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that

copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation on the rst page.

Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy

otherwise, or republish, to post on servers or to redistribute to lists, requires prior specic permission and/or a fee. Request permissions from

permissions@acm.org.

UbiComp/ISWC ’22 Adjunct, September 11–15, 2022, Cambridge, United Kingdom

ACM ISBN 978-1-4503-9423-9/22/09. . . $15.00

https://doi.org/10.1145/3544793.3560391

arXiv:2210.00888v2 [cs.LG] 12 Jan 2023

UbiComp/ISWC ’22 Adjunct, September 11–15, 2022, Cambridge, United Kingdom Liu and Suh et al.

model can be build-up, which will make signicant sense for detection of anomalies possibly relevant to well-

being [

], meanwhile, the feedback from the behavior model can, in turn, encourage users to develop a healthy

lifestyle.

Indoors is an environment where people spend much time; human activities in an indoor environment can

reect their living habits directly. Thus, indoor activity recognition has been widely used in many intelligent

systems, from smart homes and smart health to smart security [

]. For example, human activities in the kitchen

are closely associated with their dietary habits. In most cases, the frequency of open refrigerators and microwave

ovens can indicate food intake frequency in the long term. Eating too much, too frequently, or abnormal eating

time could form an unhealthy dietary habit, which can lead to many diseases like diabetes [

], obesity [

], and

cardiovascular disease [

]. Kitchen scene context-based activity recognition thus is a promising approach for diet

monitoring, and dietary treatment, also helpful for developing smart kitchens as a part of smart home [

]. In

addition, it also provides meaningful information for people to understand their dietary habits, which play an

essential role in promoting a healthy lifestyle through interventions [13].

However, compared to human activity recognition in other application scenarios, kitchen activity recognition

is still not widely explored. In this paper, we introduce a smart badge integrated with multi-modal sensors to

recognize human activity in a kitchen scenario. We designed the multi-sensor-based hardware platform to be

packaged in a light badge, and thus easily attachable to the user’s chest, and includes six dierent sensors and

two microcontrollers. It is worth mentioning that we use an infrared array sensor (thermal sensor) instead of a

camera to avoid privacy issues. In addition, we adopt a multi-channel convolutional neural network (MC-CNN)

[

] based on data and feature fusion methods and evaluate the performance of dierent sensors for activity

recognition in the kitchen.

The contributions of this work are summarized as follows:

•

A hardware platform of smart badge with 6 sensors for 14 kinds of common human activity recognition in

kitchen was proposed.

•

Two kinds of multi-channel convolutional neural network for data fusion and feature fusion are adopted

and provide high recognition accuracy of 14 kinds of human activity.

•The performances of dierent sensor in kitchen activity recognition are investigated.

The remaining of this paper is organized as follows. Section 2 presents the related works in the elds of kitchen

activity recognition and multi-modal sensor platform. The detailed presentation of hardware implementation is

introduced in Section 3. Section 4 presents the collected data and the quantitative experimental results. Finally,

Section 5 shows the analysis of the experimental results and Section 6 addresses conclusions and future works.

2 RELATED WORK

2.1 Kitchen activity recognition

Although kitchen activity recognition is not so widely explored as other application scenarios, there are still

some solutions for activity recognition in a kitchen scenario proposed over the decades as kitchen activity of

humans closely related to their dietary habits. The approaches can be grouped into vision-based and sensor-based.

Vision-based methods combined with machine learning are widely utilized for kitchen activity recognition. For

instance, Bansal et al. [

] used a dynamic SVM-HMM hybrid model to predict nine cooking activities from video

information with a recognition accuracy of 72 %. Lei et al. [

] proposed a study for ne-grained recognition

of kitchen activities with the use of RGB-D (Kinect-style) cameras, and the proposed system can robustly track

and accurately recognize detailed steps through cooking activities. Although the vision-based method has

showed a remarkable performance for HAR in dierent application scenarios [

], it often requires

high computation capability and a well-lighted environment. Besides, privacy issues also prevent its widespread

use. With the thriving development of sensor technology and pervasive computing, sensor-based HAR with

Smart-Badge: A wearable badge with multi-modal sensorsUbiComp/ISWC ’22 Adjunct, September 11–15, 2022, Cambridge, United Kingdom

privacy well protected is becomming more and more popular [

]. Luo et al. [

] demonstrated a minimal and

non-intrusive, low-power, low-cost radar-based sensing network system recognizing 15 kinds of activities with

an accuracy of 92.8 %. However, this solution lacks exibility as the sensing network system should be deployed

in the kitchen and can only detect the activity in a limited space. The wearable device has shown more exible

advantages over such distributed sensor system. Yasser et al. [

] presented a dataset for 15 kinds of kitchen

activity recognition using smartwatch accelerometers. Besides, they achieved a classication precise of 97.6

% with the use of CNN based approach, which shows that a weareable device has a great potential in kitchen

activity recognition.

2.2 Multi-modal sensors platform in human activity recognition

Compared to the limitation of computer vision technology in the human activity recognition area, like space-time

limitations, easy invasion of user privacy, and high energy consumption, the sensor-based methods with many

advantages like compact, low cost, and high computational power have become the focus of attention [

]. A

single special-purpose sensor can only recognize single series activities. It also suers from low robustness in

most cases because most sensors have limitations due to sensor deprivation, limited spatial coverage, occlusion,

imprecision, and uncertainty [

]. Besides, an unhealthy lifestyle is usually the result of much bad behavior. Thus,

using a single sensor for human activity recognition is not a perfect option in many scenarios. The concurrent use

of multiple sensors for human activity recognition provides a practical solution for complex activity recognition,

and improvement of recognition accuracy [

]. For example, Zhang et al. [

] designed a necklace with multiple

embedded sensors, such as a proximity sensor, an ambient light sensor, and an inertial measurement Unit (IMU)

sensor. The necklace can detect the eating activity more accurately after augmenting the proximity sensor data

with the ambient light and IMU sensor data. Bharti et al. [

] proposed a HuMAnsystem with ve kinds of sensors

such as IMU, temperature, air pressure, and humidity sensor, as well as Bluetooth beacon, which are deployed

on dierent parts of the body. This system showed that 21 complex activities at home could be detected with

high accuracy. Gravina et al. [

] demonstrated a system based on body-worn inertial sensors combined with

a pressure sensor to monitor in-seat activities, by which four ordinary basic emotion-relevant activities were

recognized with high accuracy. These studies about human activity recognition shows that complex human

activity can benet from multi-modal sensors information, which can increase system reliability and improve

recognition accuracy. In this paper, we design a wearable smart badge base on multiple modal sensor platform

with six dierent sensors to recognize kitchen activities, which could help users know and understand their

habits.

3 HARDWARE IMPLEMENTATION

The ergonomics and ease of use are of paramount importance for a wearable device, which can not be a burden

to the user during wearing [

]. Therefore, a smart light wearable badge was designed in this work, which can be

attached to many parts of the body exibly and easily, like shoulders, hips, and chest. As shown in Fig. 1, the

smart badge hardware system consists of four main components such as sensor module, microcontroller module,

data logger module as well as data transmission module. Six dierent sensors (IMU, optical sensor, gas sensor, air

pressure sensor, thermal IR array, and Time of Flight (ToF) ranging sensor) are connected to two microcontrollers

via the I2C interface. 791 channel data from these six sensors, including body motion and ambient information,

are sampled. Since the sample rate of these sensors varies considerably, for instance, the sample rate of IMU sensor

LSM9D01 is up to 400 Hz in fast mode, while the sample rate of gas sensor CCS811 is only 4 Hz. If all sensors are

connected to one I2C bus system, the sampling rate will be decreased dramatically. Furthermore, the number

of communication interfaces of one micro-controller is limited for connecting the six sensors separately. Two

microcontrollers (NXP iMXRT1062 based on high-performance ARM Cortex-M7 processor core and nRF52840

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

Smart-Badge:Awearablebadgewithmulti-modalsensorsforkitchenactivityrecognitionMENGXILIU,SUNGHOSUH,BOZHOU,AGNESGRÜNERBL,andPAULLUKOWICZ,Ger-manResearchCenterforArtificialIntelligence(DFKI),GermanyHumanhealthiscloselyassociatedwiththeirdailybehaviorandenvironment.However,keepingahealthylifestyleisstill...

展开>> 收起<<

Smart-Badge A wearable badge with multi-modal sensors for kitchen activity recognition.pdf

共11页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Smart-Badge A wearable badge with multi-modal sensors for kitchen activity recognition

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: