Non-parametric Bayesian annotator combination - Archive ouverte HAL Access content directly
Journal Articles Information Sciences Year : 2018

Non-parametric Bayesian annotator combination

(1, 2, 3, 4) , (3) , (3, 4)


Relying on a single imperfect human annotator is not recommended in real crowdsourced classification problems. In practice, several annotators' propositions are generally aggregated to obtain a better classification accuracy. Bayesian approaches, by modelling the relationship between each annotator's output and the possible true labels (classes), have been shown to outperform other simpler models. Unfortunately, they assume that the total number of true labels is known. This is not the case in lots of realistic scenarios such as open-world classification where the number of possible labels is undetermined and may change over time. In this paper, we show how to set a non-parametric prior over the possible label set using the Dirichlet process in order to overcome this limitation. We illustrate this prior over the Bayesian annotator combination (BAC) model from the state of the art, resulting in the so-called non-parametric BAC (NPBAC). We show how to derive its variational equations to evaluate the model and how to assess it when the Dirichlet process has a prior using the Laplace method. We apply the model to several scenarios related to closed-world classification , open-world classification and novelty detection on a dataset previously published and on two datasets related to plant classification. Our experiments show that NPBAC is able to determine the true number of labels, but also and surprisingly, it largely outperforms the parametric annotator combination by modelling more complex confusions, in particular when few or no training data are available.
Fichier principal
Vignette du fichier
parametric-bayesian-annotator-without_highlighted_corrections.pdf (484.25 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

lirmm-01703020 , version 1 (07-02-2018)



Maximilien Servajean, Romain Chailan, Alexis Joly. Non-parametric Bayesian annotator combination. Information Sciences, 2018, 436-437, pp.131-145. ⟨10.1016/j.ins.2018.01.020⟩. ⟨lirmm-01703020⟩
275 View
214 Download



Gmail Facebook Twitter LinkedIn More