TARDIS: Optimal Execution of Scientific Workflows in Apache Spark

Daniel Gaspar 1 Fabio Porto 1 Reza Akbarinia 2 Esther Pacitti 2
2 ZENITH - Scientific Data Management
LIRMM - Laboratoire d'Informatique de Robotique et de Microélectronique de Montpellier, CRISAM - Inria Sophia Antipolis - Méditerranée
Abstract : The success of using workflows for modeling large-scale scientific applications has fostered the research on parallel execution of scientific workflows in shared-nothing clusters, in which large volumes of scientific data may be stored and processed in parallel using ordinary machines. However, most of the current scientific workflow management systems do not handle the memory and data locality appropriately. Apache Spark deals with these issues by chaining activities that should be executed in a specific node, among other optimizations such as the in-memory storage of intermediate data in RDDs (Resilient Distributed Datasets). However, to take advantage of the RDDs, Spark requires existing workflows to be described using its own API, which forces the activities to be reimplemented in Python, Java, Scala or R, and this demands a big effort from the workflow programmers. In this paper, we propose a parallel scientific workflow engine called TARDIS, whose objective is to run existing workflows inside a Spark cluster, using RDDs and smart caching, in a completely transparent way for the user, i.e., without needing to reimplement the workflows in the Spark API. We evaluated our system through experiments and compared its performance with Swift/K. The results show that TARDIS performs better (up to 138% improvement) than Swift/K for parallel scientific workflow execution.
Document type :
Conference papers
Complete list of metadatas

Cited literature [13 references]  Display  Hide  Download

Contributor : Reza Akbarinia <>
Submitted on : Friday, October 20, 2017 - 10:16:38 AM
Last modification on : Thursday, February 14, 2019 - 3:50:11 PM
Long-term archiving on : Sunday, January 21, 2018 - 12:39:24 PM


Files produced by the author(s)



Daniel Gaspar, Fabio Porto, Reza Akbarinia, Esther Pacitti. TARDIS: Optimal Execution of Scientific Workflows in Apache Spark. DaWaK: Data Warehousing and Knowledge Discovery, Aug 2017, Lyon, France. pp.74-87, ⟨10.1007/978-3-319-64283-3_6⟩. ⟨lirmm-01620060⟩



Record views


Files downloads