Non-parametric Bayesian annotator combination

Maximilien Servajean 1, 2, 3, 4 Romain Chailan 3 Alexis Joly 3, 4
2 ADVANSE - ADVanced Analytics for data SciencE
LIRMM - Laboratoire d'Informatique de Robotique et de Microélectronique de Montpellier
3 ZENITH - Scientific Data Management
LIRMM - Laboratoire d'Informatique de Robotique et de Microélectronique de Montpellier, CRISAM - Inria Sophia Antipolis - Méditerranée
Abstract : Relying on a single imperfect human annotator is not recommended in real crowdsourced classification problems. In practice, several annotators' propositions are generally aggregated to obtain a better classification accuracy. Bayesian approaches, by modelling the relationship between each annotator's output and the possible true labels (classes), have been shown to outperform other simpler models. Unfortunately, they assume that the total number of true labels is known. This is not the case in lots of realistic scenarios such as open-world classification where the number of possible labels is undetermined and may change over time. In this paper, we show how to set a non-parametric prior over the possible label set using the Dirichlet process in order to overcome this limitation. We illustrate this prior over the Bayesian annotator combination (BAC) model from the state of the art, resulting in the so-called non-parametric BAC (NPBAC). We show how to derive its variational equations to evaluate the model and how to assess it when the Dirichlet process has a prior using the Laplace method. We apply the model to several scenarios related to closed-world classification , open-world classification and novelty detection on a dataset previously published and on two datasets related to plant classification. Our experiments show that NPBAC is able to determine the true number of labels, but also and surprisingly, it largely outperforms the parametric annotator combination by modelling more complex confusions, in particular when few or no training data are available.
Type de document :
Article dans une revue
Information Sciences, Elsevier, 2018, 436-437, pp.131-145. 〈10.1016/j.ins.2018.01.020〉
Liste complète des métadonnées

Littérature citée [50 références]  Voir  Masquer  Télécharger
Contributeur : Maximilien Servajean <>
Soumis le : mercredi 7 février 2018 - 13:54:30
Dernière modification le : mardi 13 février 2018 - 01:18:51


 Accès restreint
Fichier visible le : 2018-10-01

Connectez-vous pour demander l'accès au fichier




Maximilien Servajean, Romain Chailan, Alexis Joly. Non-parametric Bayesian annotator combination. Information Sciences, Elsevier, 2018, 436-437, pp.131-145. 〈10.1016/j.ins.2018.01.020〉. 〈lirmm-01703020〉



Consultations de la notice