Skip to Main content Skip to Navigation
Journal articles

Data-Centric Iteration in Dynamic Workflows

Jonas Dias 1 Gabriel Guerra 1 Fernando Rochinha 1 Alvaro Coutinho 1 Patrick Valduriez 2, 3 Marta Mattoso 1 
3 ZENITH - Scientific Data Management
LIRMM - Laboratoire d'Informatique de Robotique et de Microélectronique de Montpellier, CRISAM - Inria Sophia Antipolis - Méditerranée
Abstract : Dynamic workflows are scientific workflows to support computational science simulations, typically using dynamic processes based on runtime scientific data analyses. They require the ability of adapting the work-flow, at runtime, based on user input and dynamic steering. Supporting data-centric iteration is an important step towards dynamic workflows because user interaction with workflows is iterative. However, current sup-port for iteration in scientific workflows is static and does not allow for changing data at runtime. In this pa-per, we propose a solution based on algebraic operators and a dynamic execution model to enable workflow adaptation based on user input and dynamic steering. We introduce the concept of iteration lineage that makes provenance data management consistent with dynamic iterative workflow changes. Lineage enables scientists to interact with workflow data and configuration at runtime through an API that triggers steering. We evaluate our approach using a novel and real large-scale workflow for uncertainty quantification on a 640-core cluster. The results show impressive execution time savings from 2.5 to 24 days, compared to non-iterative workflow execution. We verify that the maximum overhead introduced by our iterative model is less than 5% of execution time. Also, our proposed steering algorithms are very efficient and run in less than 1 millisecond, in the worst-case scenario.
Complete list of metadata

Cited literature [28 references]  Display  Hide  Download
Contributor : Patrick Valduriez Connect in order to contact the contributor
Submitted on : Friday, July 29, 2016 - 9:32:00 AM
Last modification on : Friday, August 5, 2022 - 3:03:28 PM
Long-term archiving on: : Sunday, October 30, 2016 - 10:37:06 AM


Files produced by the author(s)



Jonas Dias, Gabriel Guerra, Fernando Rochinha, Alvaro Coutinho, Patrick Valduriez, et al.. Data-Centric Iteration in Dynamic Workflows. Future Generation Computer Systems, Elsevier, 2015, 46, pp.114-126. ⟨10.1016/j.future.2014.10.021⟩. ⟨lirmm-01073638⟩



Record views


Files downloads