SLO-Aware Inference Scheduler for Heterogeneous Processors in Edge Platforms

Cited 17 time in webofscience Cited 0 time in scopus
  • Hit : 516
  • Download : 146
With the proliferation of applications with machine learning (ML), the importance of edge platforms has been growing to process streaming sensor, data locally without resorting to remote servers. Such edge platforms are commonly equipped with heterogeneous computing processors such as GPU, DSP, and other accelerators, but their computational and energy budget are severely constrained compared to the data center servers. However, as an edge platform must perform the processing of multiple machine learning models concurrently for multimodal sensor data, its scheduling problem poses a new challenge to map heterogeneous machine learning computation to heterogeneous computing processors. Furthermore, processing of each input must provide a certain level of bounded response latency, making the scheduling decision critical for the edge platform. This article proposes a set of new heterogeneity-aware ML inference scheduling policies for edge platforms. Based on the regularity of computation in common ML tasks, the scheduler uses the pre-profiled behavior of each ML model and routes requests to the most appropriate processors. It also aims to satisfy the service-level objective (SLO) requirement while reducing the energy consumption for each request. For such SLO supports, the challenge of ML computation on GPUs and DSP is its inflexible preemption capability. To avoid the delay caused by a long task, the proposed scheduler decomposes a large ML task to sub-tasks by its layer in the DNN model.
Publisher
ASSOC COMPUTING MACHINERY
Issue Date
2021-07
Language
English
Article Type
Article
Citation

ACM TRANSACTIONS ON ARCHITECTURE AND CODE OPTIMIZATION, v.18, no.4, pp.1 - 26

ISSN
1544-3566
DOI
10.1145/3460352
URI
http://hdl.handle.net/10203/287096
Appears in Collection
CS-Journal Papers(저널논문)
Files in This Item
120227.pdf(1.74 MB)Download
This item is cited by other documents in WoS
⊙ Detail Information in WoSⓡ Click to see webofscience_button
⊙ Cited 17 items in WoS Click to see citing articles in records_button

qr_code

  • mendeley

    citeulike


rss_1.0 rss_2.0 atom_1.0