Parallel Computation of PDFs on Big Spatial Data Using Spark - Archive ouverte HAL Access content directly
Journal Articles Distributed and Parallel Databases Year : 2020

Parallel Computation of PDFs on Big Spatial Data Using Spark

(1) , (2) , (1) , (2) , (1)
1
2

Abstract

We consider big spatial data, which is typically produced in scientific areas such as geological or seismic interpretation. The spatial data can be produced by observation (e.g. using sensors or soil instruments) or numerical simulation programs and correspond to points that represent a 3D soil cube area. However, errors in signal processing and modeling create some uncertainty, and thus a lack of accuracy in identifying geological or seismic phenomenons. Such uncertainty must be carefully analyzed. To analyze uncertainty, the main solution is to compute a Probability Density Function (PDF) of each point in the spatial cube area. However, computing PDFs on big spatial data can be very time consuming (from several hours to even months on a computer cluster). In this paper, we propose a new solution to efficiently compute such PDFs in parallel using Spark, with three methods: data grouping, machine learning prediction and sampling. We evaluate our solution by extensive experiments on different computer clusters using big data ranging from hundreds of GB to several TB. The experimental results show that our solution scales up very well and can reduce the execution time by a factor of 33 (in the order of seconds or minutes) compared with a baseline method.
Fichier principal
Vignette du fichier
DAPDauthorVersion.pdf (2.81 Mo) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

lirmm-02045144 , version 1 (21-02-2019)

Identifiers

Cite

Ji Liu, Noel Moreno Lemus, Esther Pacitti, Fábio Porto, Patrick Valduriez. Parallel Computation of PDFs on Big Spatial Data Using Spark. Distributed and Parallel Databases, 2020, 38, pp.63-100. ⟨10.1007/s10619-019-07260-3⟩. ⟨lirmm-02045144⟩
145 View
340 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More