Cover Image
QR Code

Incremental, Iterative Data Processing with Timely Dataflow.

We describe the timely dataflow model for distributed computation and its implementation in the Naiad system. The model supports stateful iterative and incremental computations. It enables both low-latency stream processing and high-throughput batch processing, using a new approach to coordination t...

Full description

Published in:Communications of the ACM Vol. 59; no. 10; pp. 75 - 84
Main Authors: Murray, Derek G., McSherry, Frank, Isard, Michael, Isaacs, Rebecca, Barham, Paul, Abadi, Martín
Format: Article
Published: Association for Computing Machinery Oct2016
Subjects:
Data flow computing
Distributed computing
Computer systems
Batch processing
Electronic data processing
Online Access:View this record in EBSCOhost
Summary:We describe the timely dataflow model for distributed computation and its implementation in the Naiad system. The model supports stateful iterative and incremental computations. It enables both low-latency stream processing and high-throughput batch processing, using a new approach to coordination that combines asynchronous and fine-grained synchronous execution. We describe two of the programming frameworks built on Naiad: GraphLINQ for parallel graph processing, and differential dataflow for nested iterative and incremental computations. We show that a generalpurpose system can achieve performance that matches, and sometimes exceeds, that of specialized systems.