Skip to Main content Skip to Navigation
Conference papers

Provenance Data in the Machine Learning Lifecycle in Computational Science and Engineering

Abstract : Machine Learning (ML) has become essential in several industries. In Computational Science and Engineering (CSE), the complexity of the ML lifecycle comes from the large variety of data, scientists' expertise, tools, and workflows. If data are not tracked properly during the lifecycle, it becomes unfeasible to recreate a ML model from scratch or to explain to stackholders how it was created. The main limitation of prove-nance tracking solutions is that they cannot cope with provenance capture and integration of domain and ML data processed in the multiple workflows in the lifecycle, while keeping the provenance capture overhead low. To handle this problem, in this paper we contribute with a detailed characterization of provenance data in the ML lifecycle in CSE; a new provenance data representation, called PROV-ML, built on top of W3C PROV and ML Schema; and extensions to a system that tracks provenance from multiple workflows to address the characteristics of ML and CSE, and to allow for provenance queries with a standard vocabulary. We show a practical use in a real case in the O&G industry, along with its evaluation using 48 GPUs in parallel.
Document type :
Conference papers
Complete list of metadata

Cited literature [24 references]  Display  Hide  Download
Contributor : Patrick Valduriez Connect in order to contact the contributor
Submitted on : Monday, October 28, 2019 - 12:36:56 PM
Last modification on : Friday, August 5, 2022 - 3:03:28 PM
Long-term archiving on: : Wednesday, January 29, 2020 - 3:50:42 PM


Files produced by the author(s)


  • HAL Id : lirmm-02335500, version 1


Renan Souza, Leonardo Azevedo, Vítor Lourenço, Elton Soares, Raphael Thiago, et al.. Provenance Data in the Machine Learning Lifecycle in Computational Science and Engineering. WORKS 2019 - Workflows in Support of Large-Scale Science co-located with SC 2019 - ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis, Nov 2019, Denver, United States. pp.10. ⟨lirmm-02335500⟩



Record views


Files downloads