FP-Hadoop: Efficient Processing of Skewed MapReduce Jobs

Miguel Liroz-Gistau 1 Reza Akbarinia 1 Divyakant Agrawal 2 Patrick Valduriez 1
1 ZENITH - Scientific Data Management
LIRMM - Laboratoire d'Informatique de Robotique et de Microélectronique de Montpellier, CRISAM - Inria Sophia Antipolis - Méditerranée
Abstract : Nowadyas, we are witnessing the fast production of very large amount of data, particularly by the users of online systems on the Web. However, processing this big data is very challenging since both space and computational requirements are hard to satisfy. One solution for dealing with such requirements is to take advantage of parallel frameworks, such as MapReduce or Spark, that allow to make powerful computing and storage units on top of ordinary machines. Although these key-based frameworks have been praised for their high scalability and fault tolerance, they show poor performance in the case of data skew. There are important cases where a high percentage of processing in the reduce side ends up being done by only one node. In this paper, we present FP-Hadoop, a Hadoop-based system that renders the reduce side of MapReduce more parallel by efficiently tackling the problem of reduce data skew. FP-Hadoop introduces a new phase, denoted intermediate reduce (IR), where blocks of intermediate values are processed by intermediate reduce workers in parallel. With this approach, even when all intermediate values are associated to the same key, the main part of the reducing work can be performed in parallel taking benefit of the computing power of all available workers. We implemented a prototype of FP-Hadoop, and conducted extensive experiments over synthetic and real datasets. We achieved excellent performance gains compared to native Hadoop, e.g. more than 10 times in reduce time and 5 times in total execution time.
Type de document :
Article dans une revue
Information Systems, Elsevier, 2016, 60, pp.69-84. 〈10.1016/j.is.2016.03.008〉
Liste complète des métadonnées

Littérature citée [28 références]  Voir  Masquer  Télécharger

Contributeur : Reza Akbarinia <>
Soumis le : vendredi 7 octobre 2016 - 14:33:42
Dernière modification le : mercredi 7 mars 2018 - 16:23:33
Document(s) archivé(s) le : vendredi 3 février 2017 - 19:10:39


Fichiers produits par l'(les) auteur(s)



Miguel Liroz-Gistau, Reza Akbarinia, Divyakant Agrawal, Patrick Valduriez. FP-Hadoop: Efficient Processing of Skewed MapReduce Jobs. Information Systems, Elsevier, 2016, 60, pp.69-84. 〈10.1016/j.is.2016.03.008〉. 〈lirmm-01377715〉



Consultations de la notice


Téléchargements de fichiers