Skip to Main content Skip to Navigation
Journal articles

FP-Hadoop: Efficient Processing of Skewed MapReduce Jobs

Miguel Liroz-Gistau 1 Reza Akbarinia 1 Divyakant Agrawal 2 Patrick Valduriez 1 
1 ZENITH - Scientific Data Management
LIRMM - Laboratoire d'Informatique de Robotique et de Microélectronique de Montpellier, CRISAM - Inria Sophia Antipolis - Méditerranée
Abstract : Nowadyas, we are witnessing the fast production of very large amount of data, particularly by the users of online systems on the Web. However, processing this big data is very challenging since both space and computational requirements are hard to satisfy. One solution for dealing with such requirements is to take advantage of parallel frameworks, such as MapReduce or Spark, that allow to make powerful computing and storage units on top of ordinary machines. Although these key-based frameworks have been praised for their high scalability and fault tolerance, they show poor performance in the case of data skew. There are important cases where a high percentage of processing in the reduce side ends up being done by only one node. In this paper, we present FP-Hadoop, a Hadoop-based system that renders the reduce side of MapReduce more parallel by efficiently tackling the problem of reduce data skew. FP-Hadoop introduces a new phase, denoted intermediate reduce (IR), where blocks of intermediate values are processed by intermediate reduce workers in parallel. With this approach, even when all intermediate values are associated to the same key, the main part of the reducing work can be performed in parallel taking benefit of the computing power of all available workers. We implemented a prototype of FP-Hadoop, and conducted extensive experiments over synthetic and real datasets. We achieved excellent performance gains compared to native Hadoop, e.g. more than 10 times in reduce time and 5 times in total execution time.
Document type :
Journal articles
Complete list of metadata

Cited literature [28 references]  Display  Hide  Download
Contributor : Reza Akbarinia Connect in order to contact the contributor
Submitted on : Friday, October 7, 2016 - 2:33:42 PM
Last modification on : Friday, August 5, 2022 - 3:03:28 PM
Long-term archiving on: : Friday, February 3, 2017 - 7:10:39 PM


Files produced by the author(s)



Miguel Liroz-Gistau, Reza Akbarinia, Divyakant Agrawal, Patrick Valduriez. FP-Hadoop: Efficient Processing of Skewed MapReduce Jobs. Information Systems, Elsevier, 2016, 60, pp.69-84. ⟨10.1016/⟩. ⟨lirmm-01377715⟩



Record views


Files downloads