An Efficient Solution for Processing Skewed MapReduce Jobs

Reza Akbarinia 1 Miguel Liroz-Gistau 1 Divyakant Agrawal 2 Patrick Valduriez 3, 1
1 ZENITH - Scientific Data Management
LIRMM - Laboratoire d'Informatique de Robotique et de Microélectronique de Montpellier, CRISAM - Inria Sophia Antipolis - Méditerranée
Abstract : Although MapReduce has been praised for its high scalability and fault tolerance, it has been criticized in some points, in particular, its poor performance in the case of data skew. There are important cases where a high percentage of processing in the reduce side is done by a few nodes, or even one node, while the others remain idle. There have been some attempts to address the problem of data skew, but only for specific cases. In particular, there is no proposed solution for the cases where most of the intermediate values correspond to a single key, or when the number of keys is less than the number of reduce workers. In this paper, we propose FP-Hadoop, a system that makes the reduce side of MapReduce more parallel, and efficiently deals with the problem of data skew in the reduce side. In FP-Hadoop, there is a new phase, called intermediate reduce (IR), in which blocks of intermediate values, constructed dynamically, are processed by intermediate reduce workers in parallel, by using a scheduling strategy. By using the IR phase, even if all intermediate values belong to only one key, the main part of the reducing work can be done in parallel by using the computing resources of all available workers. We implemented a prototype of FP-Hadoop, and conducted extensive experiments over synthetic and real datasets. We achieved excellent performance gains compared to native Hadoop, e.g. more than 10 times in reduce time and 5 times in total execution time.
Type de document :
Communication dans un congrès
Globe'2015: 8th International Conference on Data Management in Cloud, Grid and P2P Systems, Sep 2015, Valencia, Spain
Liste complète des métadonnées
Contributeur : Reza Akbarinia <>
Soumis le : mercredi 10 juin 2015 - 11:47:32
Dernière modification le : samedi 27 janvier 2018 - 01:31:47
Document(s) archivé(s) le : mardi 25 avril 2017 - 06:18:12


Fichiers produits par l'(les) auteur(s)


  • HAL Id : lirmm-01162359, version 1


Reza Akbarinia, Miguel Liroz-Gistau, Divyakant Agrawal, Patrick Valduriez. An Efficient Solution for Processing Skewed MapReduce Jobs. Globe'2015: 8th International Conference on Data Management in Cloud, Grid and P2P Systems, Sep 2015, Valencia, Spain. 〈lirmm-01162359〉



Consultations de la notice


Téléchargements de fichiers