Raw data queries during data-intensive parallel workflow execution

Abstract : Computer simulations consume and produce huge amounts of raw data files presented in different formats, e.g., HDF5 in computational fluid dynamics simulations. Users often need to analyze domain-specific data based on related data elements from multiple files during the execution of computer simulations. In a raw data analysis, one should identify regions of interest in the data space and retrieve the content of specific related raw data files. Existing solutions, such as FastBit and RAW, are limited to a single raw data file analysis and can only be used after the execution of computer simulations. Scientific Workflow Management Systems (SWMS) can manage the dataflow of computer simulations and register related raw data files at a provenance database. This paper aims to combine the advantages of a dataflow-aware SWMS and the raw data file analysis techniques to allow for queries on raw data file elements that are related, but reside in separate files. We propose a component-based architecture, named as ARMFUL (Analysis of Raw data from Multiple Files) with raw data extraction and indexing techniques, which allows for a direct access to specific elements or regions of raw data space. ARMFUL innovates by using a SWMS provenance database to add a dataflow access path to raw data files. ARMFUL facilitates the invocation of ad-hoc programs and third party tools (e.g., FastBit tool) for raw data analyses. In our experiments, a real parallel computational fluid dynamics is executed, exploring different alternatives of raw data extraction, indexing and analysis.
Type de document :
Article dans une revue
Future Generation Computer Systems, Elsevier, 2017, 75, pp.402-422. 〈10.1016/j.future.2017.01.016〉
Liste complète des métadonnées

Littérature citée [40 références]  Voir  Masquer  Télécharger

Contributeur : Patrick Valduriez <>
Soumis le : mardi 24 janvier 2017 - 16:24:04
Dernière modification le : jeudi 24 mai 2018 - 15:59:21
Document(s) archivé(s) le : mardi 25 avril 2017 - 17:46:16


paper-FGCS-2016-author version...
Fichiers produits par l'(les) auteur(s)




Vítor Silva, José Leite, José Camata, Daniel De Oliveira, Alvaro Coutinho, et al.. Raw data queries during data-intensive parallel workflow execution. Future Generation Computer Systems, Elsevier, 2017, 75, pp.402-422. 〈10.1016/j.future.2017.01.016〉. 〈lirmm-01445219〉



Consultations de la notice


Téléchargements de fichiers