Controlling Utterance Length in NMT-based Word Segmentation with Attention

Pierre Godard 1, 2 Laurent Besacier 2 François Yvon 1
1 TLP - Traitement du Langage Parlé
LIMSI - Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur : 247329
Abstract : One of the basic tasks of computational language documentation (CLD) is to identify word boundaries in an unsegmented phonemic stream. While several unsupervised monolingual word segmentation algorithms exist in the literature, they are challenged in real-world CLD settings by the small amount of available data. A possible remedy is to take advantage of glosses or translation in a foreign, well-resourced, language, which often exist for such data. In this paper, we explore and compare ways to exploit neural machine translation models to perform unsupervised boundary detection with bilingual information, notably introducing a new loss function for jointly learning alignment and segmentation. We experiment with an actual under-resourced language, Mboshi, and show that these techniques can effectively control the output segmentation length.
Complete list of metadatas

Cited literature [60 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-02343206
Contributor : Limsi Publications <>
Submitted on : Saturday, November 2, 2019 - 3:07:56 PM
Last modification on : Wednesday, November 6, 2019 - 1:26:46 AM

File

IWSLT2019_paper_5.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02343206, version 1

Citation

Pierre Godard, Laurent Besacier, François Yvon. Controlling Utterance Length in NMT-based Word Segmentation with Attention. International Workshop on Spoken Language Translation, Nov 2019, Hong-Kong, China. ⟨hal-02343206⟩

Share

Metrics

Record views

25

Files downloads

9