Skip to Main content Skip to Navigation

Data reduction in scientific workflows using provenance monitoring and user steering

Abstract : Scientific workflows need to be iteratively, and often interactively, executed for large input datasets. Reducing data from input datasets is a powerful way to reduce overall execution time in such workflows. When this is accomplished online (i.e., without requiring the user to stop execution to reduce the data, and then resume), it can save much time. However, determining which subsets of the input data should be removed becomes a major problem. A related problem is to guarantee that the workflow system will maintain execution and data consistent with the reduction. Keeping track of how users interact with the workflow is essential for data provenance purposes. In this paper, we adopt the " human-in-the-loop " approach, which enables users to steer the running workflow and reduce subsets from datasets online. We propose an adaptive workflow monitoring approach that combines provenance data monitoring and computational steering to support users in analyzing the evolution of key parameters and determining the subset of data to remove. We extend a provenance data model to keep track of users' interactions when they reduce data at runtime. In our experimental validation, we develop a test case from the oil and gas domain, using a 936-cores cluster. The results on this test case show that the approach yields reductions of 32% of execution time and 14% of the data processed.
Complete list of metadatas

https://hal-lirmm.ccsd.cnrs.fr/lirmm-01679967
Contributor : Patrick Valduriez <>
Submitted on : Wednesday, January 10, 2018 - 12:14:27 PM
Last modification on : Tuesday, March 17, 2020 - 3:40:30 AM

File

FUTURE_3820_ack.pdf
Files produced by the author(s)

Identifiers

Collections

Citation

Renan Souza, Vitor Silva, Alvaro L.G.A. Coutinho, Patrick Valduriez, Marta Mattoso. Data reduction in scientific workflows using provenance monitoring and user steering. Future Generation Computer Systems, Elsevier, In press, ⟨10.1016/j.future.2017.11.028⟩. ⟨lirmm-01679967⟩

Share

Metrics

Record views

320

Files downloads

329