Rateless Codes for Near-Perfect Load Balancing in Distributed Matrix-Vector Multiplication.
Large-scale machine learning and data mining applications require computer systems to perform massive matrix-vector and matrix-matrix multiplication operations that need to be parallelized across multiple nodes. The presence of straggling nodes--computing nodes that unpredictably slow down or fail--...
| Published in: | Communications of the ACM Vol. 65; no. 5; pp. 111 - 119 |
|---|---|
| Main Authors: | , , , , |
| Format: | Article |
| Published: |
Association for Computing Machinery
May2022
|
| Subjects: | |
| Online Access: | View this record in EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=hlh&AN=156514001&site=ehost-live header: @attributes: shortDbName: hlh uiTerm: 156514001 longDbName: Humanities International Complete uiTag: AN controlInfo: bkinfo: jinfo: jid: 00010782 ACM jtl: Communications of the ACM issn: 00010782 maglogo: N pubinfo: dt: May2022 vid: 65 iid: 5 pid: 68 pub: Association for Computing Machinery artinfo: ui: 156514001 10.1145/3524298 ppf: 111 ppct: 8 formats: tig: atl: Rateless Codes for Near-Perfect Load Balancing in Distributed Matrix-Vector Multiplication. aug: au: Mallick, Ankur Chaudhari, Malhar Sheth, Utsav Palanikumar, Ganesh Joshi, Gauri affil: Large-scale machine learning and data mining applications require computer systems to perform massive matrix-vector and matrix-matrix multiplication operations that need to be parallelized across multiple nodes. The presence of straggling nodes--computing nodes that unpredictably slow down or fail--is a major bottleneck in such distributed computations. Ideal load balancing strategies that dynamically allocate more tasks to faster nodes require knowledge or monitoring of node speeds as well as the ability to quickly move data. Recently proposed fixed-rate erasure coding strategies can handle unpredictable node slowdown, but they ignore partial work done by straggling nodes, thus resulting in a lot of redundant computation. We propose a rateless fountain coding strategy that achieves the best of both worlds--we prove that its latency is asymptotically equal to ideal load balancing, and it performs asymptotically zero redundant computations. Our idea is to create linear combinations of the m rows of the matrix and assign these encoded rows to different worker nodes. The original matrix-vector product can be decoded as soon as slightly more than m rowvector products are collectively finished by the nodes. Evaluation on parallel and distributed computing yields as much as three times speedup over uncoded schemes Oracle Corporation, Redwood City, CA, USA Automation Anywhere, San Jose, CA, USA Apple Inc., Cupertino, CA, USA su: Load balancing (Computer networks) Computer programming Machine learning Data mining sug: subj: Load balancing (Computer networks) Computer programming Machine learning Data mining ab: Large-scale machine learning and data mining applications require computer systems to perform massive matrix-vector and matrix-matrix multiplication operations that need to be parallelized across multiple nodes. The presence of straggling nodes--computing nodes that unpredictably slow down or fail--is a major bottleneck in such distributed computations. Ideal load balancing strategies that dynamically allocate more tasks to faster nodes require knowledge or monitoring of node speeds as well as the ability to quickly move data. Recently proposed fixed-rate erasure coding strategies can handle unpredictable node slowdown, but they ignore partial work done by straggling nodes, thus resulting in a lot of redundant computation. We propose a rateless fountain coding strategy that achieves the best of both worlds--we prove that its latency is asymptotically equal to ideal load balancing, and it performs asymptotically zero redundant computations. Our idea is to create linear combinations of the m rows of the matrix and assign these encoded rows to different worker nodes. The original matrix-vector product can be decoded as soon as slightly more than m rowvector products are collectively finished by the nodes. Evaluation on parallel and distributed computing yields as much as three times speedup over uncoded schemes. pubtype: Periodical doctype: Article src: R language: English refInfo: copyright: @attributes: flag: Y dt: @attributes: year: 2022 holdings: @attributes: islocal: N |
|---|