Algebraic Dataflows for Big Data Analysis

Abstract : Analyzing big data requires the support of dataflows with many activities to extract and explore relevant information from the data. Recent approaches such as Pig Latin propose a high-level language to model such dataflows. However, the dataflow execution is typically delegated to a MapReduce implementation such as Hadoop, which does not follow an algebraic approach, thus it cannot take advantage of the optimization opportunities of PigLatin algebra. In this paper, we propose an approach for big data analysis based on algebraic workflows, which yields optimization and parallel execution of activities and supports user steering using provenance queries. We illustrate how a big data processing dataflow can be modeled using the algebra. Through an experimental evaluation using real datasets and the execution of the dataflow with Chiron, an engine that supports our algebra, we show that our approach yields performance gains of up to 19.6% using algebraic optimizations in the dataflow and up to 39.1% of time saved on a user steering scenario.
Complete list of metadatas

https://hal-lirmm.ccsd.cnrs.fr/lirmm-00857221
Contributor : Patrick Valduriez <>
Submitted on : Monday, September 9, 2013 - 5:53:12 PM
Last modification on : Friday, March 15, 2019 - 1:15:01 AM

Identifiers

Collections

Citation

Dias Jonas, Eduardo Ogasawara, Oliveira Daniel De, Fabio Porto, Patrick Valduriez, et al.. Algebraic Dataflows for Big Data Analysis. IEEE BigData, Oct 2013, Santa Clara, United States. pp.6, ⟨10.1109/BigData.2013.6691567⟩. ⟨lirmm-00857221⟩

Share

Metrics

Record views

1006