Energy Measurement System for Data Lake - IRIT - Université Toulouse 1 Capitole
Communication Dans Un Congrès Année : 2024

Energy Measurement System for Data Lake

Résumé

Data Lakes are increasingly deployed as a solution for Big Data analytics. Recent improvements in Data Lake technology have focused on improving data access, governance, and discoverability. However, the energy consumption of data operations, a non-trivial issue for eco-conscious organizations, is currently overlooked. Furthermore, existing monitoring tools do not adequately address the complexities of Data Lake architectures. This paper presents the initial phase of developing a system for measuring energy in Data Lake pipeline operations. The novelty of our solution lies in the fact that we define four measures to assess the power usage of crucial hardware components in a Data Lake context: CPU, RAM, NIC, and storage devices. To validate our approach, we developed a monitoring tool grounded in real-world datasets from a Data Lake benchmark.
Fichier principal
Vignette du fichier
ACIIDS_2024_paper_63-1.pdf (678.74 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04549466 , version 1 (17-04-2024)

Identifiants

  • HAL Id : hal-04549466 , version 1

Citer

Philippe Roose, Hernán H Álvarez Valera, Alexandre Maurice, Franck Ravat, Jiefu Song, et al.. Energy Measurement System for Data Lake. ACIIDS 2024 - 16th Asian Conference on Intelligent Information and Database Systems, Apr 2024, Ras Al Khaimah, United Arab Emirates. à paraître. ⟨hal-04549466⟩
634 Consultations
100 Téléchargements

Partager

More