Data reduction in scientific workflows using provenance monitoring and user steering - LIRMM - Laboratoire d’Informatique, de Robotique et de Microélectronique de Montpellier
Journal Articles Future Generation Computer Systems Year : 2020

Data reduction in scientific workflows using provenance monitoring and user steering

Abstract

Scientific workflows need to be iteratively, and often interactively, executed for large input datasets. Reducing data from input datasets is a powerful way to reduce overall execution time in such workflows. When this is accomplished online (i.e., without requiring the user to stop execution to reduce the data, and then resume), it can save much time. However, determining which subsets of the input data should be removed becomes a major problem. A related problem is to guarantee that the workflow system will maintain execution and data consistent with the reduction. Keeping track of how users interact with the workflow is essential for data provenance purposes. In this paper, we adopt the " human-in-the-loop " approach, which enables users to steer the running workflow and reduce subsets from datasets online. We propose an adaptive workflow monitoring approach that combines provenance data monitoring and computational steering to support users in analyzing the evolution of key parameters and determining the subset of data to remove. We extend a provenance data model to keep track of users' interactions when they reduce data at runtime. In our experimental validation, we develop a test case from the oil and gas domain, using a 936-cores cluster. The results on this test case show that the approach yields reductions of 32% of execution time and 14% of the data processed.
Fichier principal
Vignette du fichier
FUTURE_3820_ack.pdf (3.84 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

lirmm-01679967 , version 1 (10-01-2018)

Identifiers

Cite

Renan Souza, Vitor Silva, Alvaro Luiz Gayoso de Azeredo Coutinho, Patrick Valduriez, Marta Mattoso. Data reduction in scientific workflows using provenance monitoring and user steering. Future Generation Computer Systems, 2020, 110, pp.481-501. ⟨10.1016/j.future.2017.11.028⟩. ⟨lirmm-01679967⟩
370 View
581 Download

Altmetric

Share

More