FP-Hadoop: Efficient Execution of Parallel Jobs Over Skewed Data - LIRMM - Laboratoire d’Informatique, de Robotique et de Microélectronique de Montpellier Access content directly
Journal Articles Proceedings of the VLDB Endowment (PVLDB) Year : 2015

FP-Hadoop: Efficient Execution of Parallel Jobs Over Skewed Data

Miguel Liroz-Gistau
  • Function : Author
  • PersonId : 901689
Reza Akbarinia


Big data parallel frameworks, such as MapReduce or Spark have been praised for their high scalability and performance, but show poor performance in the case of data skew. There are important cases where a high percentage of processing in the reduce side ends up being done by only one node. In this demonstration, we illustrate the use of FP-Hadoop, a system that efficiently deals with data skew in MapReduce jobs. In FP-Hadoop, there is a new phase, called intermediate reduce (IR), in which blocks of intermediate values , constructed dynamically, are processed by intermediate reduce workers in parallel, by using a scheduling strategy. Within the IR phase, even if all intermediate values belong to only one key, the main part of the reducing work can be done in parallel using the computing resources of all available workers. We implemented a prototype of FP-Hadoop, and conducted extensive experiments over synthetic and real datasets. We achieve excellent performance gains compared to native Hadoop, e.g. more than 10 times in reduce time and 5 times in total execution time. During our demonstration, we give the users the possibility to execute and compare job executions in FP-Hadoop and Hadoop. They can retrieve general information about the job and the tasks and a summary of the phases. They can also visually compare different configurations to explore the difference between the approaches.
Fichier principal
Vignette du fichier
vldb_2015.pdf (483.65 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

lirmm-01162362 , version 1 (10-06-2015)



Miguel Liroz-Gistau, Reza Akbarinia, Patrick Valduriez. FP-Hadoop: Efficient Execution of Parallel Jobs Over Skewed Data. Proceedings of the VLDB Endowment (PVLDB), 2015, 8 (12), pp.1856-1867. ⟨10.14778/2824032.2824085⟩. ⟨lirmm-01162362⟩
407 View
727 Download



Gmail Mastodon Facebook X LinkedIn More