Non-parametric Bayesian annotator combination

Maximilien Servajean 1, 2, 3, 4 Romain Chailan 3 Alexis Joly 3, 4
2 ADVANSE - ADVanced Analytics for data SciencE
LIRMM - Laboratoire d'Informatique de Robotique et de Microélectronique de Montpellier
3 ZENITH - Scientific Data Management
LIRMM - Laboratoire d'Informatique de Robotique et de Microélectronique de Montpellier, CRISAM - Inria Sophia Antipolis - Méditerranée
Abstract : Relying on a single imperfect human annotator is not recommended in real crowdsourced classification problems. In practice, several annotators' propositions are generally aggregated to obtain a better classification accuracy. Bayesian approaches, by modelling the relationship between each annotator's output and the possible true labels (classes), have been shown to outperform other simpler models. Unfortunately, they assume that the total number of true labels is known. This is not the case in lots of realistic scenarios such as open-world classification where the number of possible labels is undetermined and may change over time. In this paper, we show how to set a non-parametric prior over the possible label set using the Dirichlet process in order to overcome this limitation. We illustrate this prior over the Bayesian annotator combination (BAC) model from the state of the art, resulting in the so-called non-parametric BAC (NPBAC). We show how to derive its variational equations to evaluate the model and how to assess it when the Dirichlet process has a prior using the Laplace method. We apply the model to several scenarios related to closed-world classification , open-world classification and novelty detection on a dataset previously published and on two datasets related to plant classification. Our experiments show that NPBAC is able to determine the true number of labels, but also and surprisingly, it largely outperforms the parametric annotator combination by modelling more complex confusions, in particular when few or no training data are available.
Document type :
Journal articles
Complete list of metadatas

Cited literature [24 references]  Display  Hide  Download

https://hal-lirmm.ccsd.cnrs.fr/lirmm-01703020
Contributor : Maximilien Servajean <>
Submitted on : Wednesday, February 7, 2018 - 1:54:30 PM
Last modification on : Thursday, June 6, 2019 - 2:45:35 PM

File

parametric-bayesian-annotator-...
Files produced by the author(s)

Identifiers

Citation

Maximilien Servajean, Romain Chailan, Alexis Joly. Non-parametric Bayesian annotator combination. Information Sciences, Elsevier, 2018, 436-437, pp.131-145. ⟨10.1016/j.ins.2018.01.020⟩. ⟨lirmm-01703020⟩

Share

Metrics

Record views

435

Files downloads

99