EC-NAS ENERGY CONSUMPTION A WARE TABULAR BENCHMARKS FOR NEURAL ARCHITECTURE SEARCH Pedram BakhtiarifardChristian IgelRaghavendra Selvan

2025-04-26 0 0 6.51MB 9 页 10玖币

侵权投诉

EC-NAS: ENERGY CONSUMPTION AWARE TABULAR BENCHMARKS FOR NEURAL

ARCHITECTURE SEARCH

Pedram Bakhtiarifard†Christian Igel†Raghavendra Selvan†∗

†Department of Computer Science, University of Copenhagen, Denmark

ABSTRACT

Energy consumption from the selection, training, and deploy-

ment of deep learning models has seen a signiﬁcant uptick

recently. This work aims to facilitate the design of energy-

efﬁcient deep learning models that require less computa-

tional resources and prioritize environmental sustainability

by focusing on the energy consumption. Neural architecture

search (NAS) beneﬁts from tabular benchmarks, which eval-

uate NAS strategies cost-effectively through pre-computed

performance statistics. We advocate for including energy

efﬁciency as an additional performance criterion in NAS.

To this end, we introduce an enhanced tabular benchmark

encompassing data on energy consumption for varied ar-

chitectures. The benchmark, designated as EC-NAS1, has

been made available in an open-source format to advance

research in energy-conscious NAS. EC-NAS incorporates a

surrogate model to predict energy consumption, aiding in di-

minishing the energy expenditure of the dataset creation. Our

ﬁndings emphasize the potential of EC-NAS by leveraging

multi-objective optimization algorithms, revealing a balance

between energy usage and accuracy. This suggests the feasi-

bility of identifying energy-lean architectures with little or no

compromise in performance.

Index Terms—Energy-aware benchmark, neural archi-

tecture search, sustainable machine learning, multi-objective

optimization

1. INTRODUCTION

Neural Architecture Search (NAS) strategies, which explore

model architectures based on training and evaluation metrics,

have demonstrated their ability to reveal novel designs with

state-of-the-art performance [1,2,3]. While promising, NAS

comes with computational and energy-intensive demands,

leading to signiﬁcant environmental concerns due to the car-

bon footprint incurred due to energy consumption [4,5,6].

Given the rapidly increasing computational requirements of

deep learning models [7], there is an imperative to address

the balance between performance and resource efﬁciency.

∗The authors acknowledge funding received under European Union’s

Horizon Europe Research and Innovation programme under grant agreements

No. 101070284 and No. 101070408.

1Source code is available at: https://github.com/saintslab/

EC-NAS-Bench

Fig. 1. Scatter plot of about 423k CNN architectures showing

training energy (E) vs. validation performance (Pv) across

four training budgets. Solutions in the top-right (red ellipse)

prioritize performance at high energy costs. Joint optimiza-

tion shifts preferred solutions to the left (green ellipse), indi-

cating reduced energy with minimal performance loss.

Efﬁcient evaluation of NAS strategies has gained traction,

using pre-computed performance statistics in tabular bench-

marks and the use of surrogate and one-shot models [8,9,10,

2,11]. Nevertheless, the primary focus remains on perfor-

mance, with the trade-offs between performance and energy

efﬁciency often overlooked. This trade-off is visually repre-

sented in Figure 1, illustrating the potential to ﬁnd energy-

efﬁcient models without compromising performance. Align-

ing with recent advancements in energy-aware NAS research,

we advocate for integrating energy consumption as a pivotal

metric in tabular NAS benchmarks. We aim to uncover inher-

ently efﬁcient deep learning models, leveraging pre-computed

energy statistics for sustainable model discovery. This per-

spective is supported by recent works, such as the EA-HAS-

Bench [12], which emphasizes the trade-offs between perfor-

mance and energy consumption. Furthermore, the diverse ap-

plications of NAS in areas like speech emotion recognition

[13] and visual-inertial odometry [14] underscore its versatil-

ity and the need for efﬁciency.

arXiv:2210.06015v4 [cs.LG] 22 Mar 2024

2. ENERGY AWARENESS IN NAS

Building upon the foundational NAS-Bench-101 [10], we

introduce our benchmark, EC-NAS, to accentuate the im-

perative of energy efﬁciency in NAS. Our adaptation of this

dataset, initially computed using an exorbitant 100 TPU

years equivalent of compute time, serves our broader mission

of steering NAS methodologies towards energy consumption

awareness.

2.1. Architectural Design and Blueprint

Central to our method are architectures tailored for CIFAR-

10 image classiﬁcation [15]. We introduce additional objec-

tives for emphasizing the signiﬁcance of hardware-speciﬁc

efﬁciency trends in deep learning models. The architec-

tural space is conﬁned to the topological space of cells,

with each cell being a conﬁgurable feedforward network.

In terms of cell encoding, these individual cells are rep-

resented as directed acyclic graphs (DAGs). Each DAG,

G(V, M), has N=|V|vertices (or nodes) and edges de-

scribed in a binary adjacency matrix M∈ {0,1}N×N.

The set of operations (labels) that each node can realise

is given by L′={input,output} ∪ L, where L=

{3x3conv,1x1conv,3x3maxpool}. Two of the Nnodes

are always ﬁxed as input and output to the network. The

remaining N−2nodes can take up one of the labels in L. The

connections between nodes of the DAG are encoded in the

upper-triangular adjacency matrix with no self-connections

(zero main diagonal entries). For a given architecture, A,

every entry αi,j ∈MAdenotes an edge, from node ito node

jwith operations i, j ∈ L and its labelled adjacency matrix,

LA∈MA× L′.

2.2. Energy Measures in NAS

Traditional benchmarks, while insightful, often fall short

of providing a complete energy consumption proﬁle. In

EC-NAS, we bring the signiﬁcance of energy meaures to

the forefront, crafting a comprehensive view that synthesizes

both hardware and software intricacies. The mainstays of

neural network training – GPUs and TPUs – are notorious

for their high energy consumption [6,16]. To capture these

nuances, we utilize and adopt the Carbontracker tool

[6] to our speciﬁc needs, allowing us to observe total energy

costs, computational times, and aggregate carbon footprints.

2.3. Surrogate Model for Energy Estimation

The landscape of NAS has transformed to encompass a

broader spectrum of metrics. Energy consumption, pivotal

during model training, offers insights beyond the purview

of traditional measures such as ﬂoating-point operations

(FPOPs) and computational time. Given the variability in

computational time, owing to diverse factors like parallel

infrastructure, this metric can occasionally be misleading.

5 10 15 20 25 30 35 40

Actual Energy (kWh)

Predicted Energy (kWh)

Kendall-Tau R2= 0.9030

0 500 1000 1500 2000 2500 3000

No. of training datapoints

0.5

1.0

1.5

2.0

2.5

3.0

MAE on ﬁxed test set

Fig. 2. Scatter plot depicting the Kendall-Tau correlation co-

efﬁcient between predicted and actual energy consumption

(left) and the inﬂuence of training data size on test accuracy

(right). Error bars are based on 10 random initializations.

Energy consumption, in contrast, lends itself as a more con-

sistent and comprehensive measure, factoring in software and

hardware variations. We measure the energy consumption of

training the architectures on the CIFAR-10 dataset, follow-

ing the protocols to NAS-Bench-101. The in-house SLURM

cluster, powered by an NVIDIA Quadro RTX 6000 GPU and

two Intel CPUs, provides an optimal environment.

The vast architecture space, however, introduces chal-

lenges in the direct energy estimation. Our remedy to this

is a surrogate model approach, wherein we derived insights

to guide a multi-layer perceptron (MLP) model by training

using a representative subset of architectures. This surrogate

model adeptly predicts energy consumption patterns, bridg-

ing computational demand and energy efﬁciency. Its efﬁcacy

is highlighted by the strong correlation between its predic-

tions and actual energy consumption values, as illustrated in

Figure 2.

2.4. Dataset Analysis and Hardware Consistency

Understanding architectural characteristics and the trade-offs

they introduce is crucial. This involves studying operations,

their impacts on efﬁciency and performance, as well as the

overarching inﬂuence of hardware on energy costs. Training

time and energy consumption trends naturally increase with

model size. However, gains in performance tend to plateau for

models characterized by larger DAGs. Interestingly, while pa-

rameter variation across model sizes remains minimal, train-

ing time and energy consumption show more signiﬁcant vari-

ability for more extensive models. These ﬁndings highlight

the multifaceted factors affecting performance and efﬁciency.

Different operations can also have a profound impact on

performance. For instance, speciﬁc operation replacements

signiﬁcantly boost validation accuracy while increasing en-

ergy consumption without increasing training time. This

complex relationship between training time, energy con-

sumption and performance underscore the importance of a

comprehensive approach in NAS. The impact of swapping

one operation for another on various metrics, including en-

ergy consumption, training time, validation accuracy, and

parameter count, is captured in Figure 3.

In EC-NAS, we further probed the energy consumption

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

EC-NAS:ENERGYCONSUMPTIONAWARETABULARBENCHMARKSFORNEURALARCHITECTURESEARCHPedramBakhtiarifard†ChristianIgel†RaghavendraSelvan†∗†DepartmentofComputerScience,UniversityofCopenhagen,DenmarkABSTRACTEnergyconsumptionfromtheselection,training,anddeploy-mentofdeeplearningmodelshasseenasignificantuptickrecen...

展开>> 收起<<

EC-NAS ENERGY CONSUMPTION A WARE TABULAR BENCHMARKS FOR NEURAL ARCHITECTURE SEARCH Pedram BakhtiarifardChristian IgelRaghavendra Selvan.pdf

共9页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

EC-NAS ENERGY CONSUMPTION A WARE TABULAR BENCHMARKS FOR NEURAL ARCHITECTURE SEARCH Pedram BakhtiarifardChristian IgelRaghavendra Selvan

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: