Skip to Main content Skip to Navigation
Journal articles

Data reduction in scientific workflows using provenance monitoring and user steering

Abstract : Scientific workflows need to be iteratively, and often interactively, executed for large input datasets. Reducing data from input datasets is a powerful way to reduce overall execution time in such workflows. When this is accomplished online (i.e., without requiring the user to stop execution to reduce the data, and then resume), it can save much time. However, determining which subsets of the input data should be removed becomes a major problem. A related problem is to guarantee that the workflow system will maintain execution and data consistent with the reduction. Keeping track of how users interact with the workflow is essential for data provenance purposes. In this paper, we adopt the " human-in-the-loop " approach, which enables users to steer the running workflow and reduce subsets from datasets online. We propose an adaptive workflow monitoring approach that combines provenance data monitoring and computational steering to support users in analyzing the evolution of key parameters and determining the subset of data to remove. We extend a provenance data model to keep track of users' interactions when they reduce data at runtime. In our experimental validation, we develop a test case from the oil and gas domain, using a 936-cores cluster. The results on this test case show that the approach yields reductions of 32% of execution time and 14% of the data processed.
Complete list of metadata
Contributor : Patrick Valduriez Connect in order to contact the contributor
Submitted on : Wednesday, January 10, 2018 - 12:14:27 PM
Last modification on : Friday, October 22, 2021 - 3:07:18 PM


Files produced by the author(s)



Renan Souza, Vitor Silva, Alvaro Luiz Gayoso de Azeredo Coutinho, Patrick Valduriez, Marta Mattoso. Data reduction in scientific workflows using provenance monitoring and user steering. Future Generation Computer Systems, Elsevier, 2020, 110, pp.481-501. ⟨10.1016/j.future.2017.11.028⟩. ⟨lirmm-01679967⟩



Record views


Files downloads