Sign Language Computational Linguistics

About this research line

We research sign language processing in collaboration with (among others) the Flemish Sign Language community, to co-create AI-driven sign language technology.

In Flanders, only about 13,000 people can communicate in Flemish Sign Language (Vlaamse Gebarentaal, VGT). For many of those people, VGT is their preferred language. Since most hearing people do not understand sign language, signers and non-signers mostly communicate through interpreters or through written language. Neither is practical for ad-hoc or day-to-day interaction, or for getting to know each other on an informal basis. Interpreters are only available by appointment and need to be paid, and not all signers are equally fluent in written communication.

In recent years, AI-powered tools like ChatGPT, Gemini and Whisper have revolutionised spoken language communication. However, sign language technology lags behind for several reasons. Data scarcity, the limited level of standardisation, the fragmentation of sign language communities, the often highly improvisational nature of sign languages and the relatively small user base all contribute to this. At IDLab AIRO, we directly address some of these factors that hinder sign language technology development.

Our recent results build on the pioneering work of our alumni, Dr Lionel Pigou and Dr Mathieu De Coster, whose research focused on automatically recognising individual signs. This culminated in the co-creation, with the Flemish Sign Language Centre (VGTC), of SignBuddy — an AI-powered data collection tool for isolated signs ‘in the wild’. The collected data was used to validate and refine a scalable, large dictionary search method, which was later released into the Flemish Sign Language Dictionary (https://woordenboek.vlaamsegebarentaal.be/) as the world’s first publicly available sign-to-text dictionary search. Users of the VGT dictionary can now search through 12,000 signs for translations simply by performing any sign. Both SignBuddy and the sign-to-text search in the VGT dictionary are successful outcomes of our continuous co-creation process with VGTC. This collaboration ensures community requirements and preferences are safeguarded throughout.

More recently, our research focus has shifted from proficient sign recognition to sign language computational linguistics, and more specifically to understanding the independent sign parameters – phonemes – that constitute sign language, and their representation in a sign language technology context. However, the same vision still drives our research: producing useful and meaningful outputs for the Deaf and Hard of Hearing communities in the short term, which can serve as stepping stones for larger developments in the long term. This fosters community-driven research and preserves the co-creation relationships that are fundamental to our approach.

Active researchers

Related publications

Grounding sign language representation learning in phonology

Toon Vandendriessche, Mathieu De Coster, Joni Dambre

In Proceedings 12th Workshop on the Representation and Processing of Sign Languages : Language in Motion 2026

BIBLIO

Abstract

Sign language recognition systems are commonly trained using gloss-level supervision, treating signs as holistic lexical units. While effective for classification, such approaches entangle sub-lexical structure and fail to capture the phonological parameters that govern sign formation, limiting interpretability, robustness, and cross-lingual transfer. In this work, we propose a phonologically informed representation learning architecture that explicitly structures the latent space according to linguistic principles. Grounded in the Dependency Model – a phonological model used to describe Flemish Sign Language (VGT) – our hierarchical architecture disentangles parameter-specific subspaces for handshape and location and is trained with multi-label phoneme supervision. To evaluate whether phonological information is directly encoded in the geometry of the embedding space, we introduce a non-parametric probing method that measures neighbourhood consistency across increasing scales. We show that conventional gloss-based networks achieve reasonable performance only for very small neighbourhoods, reflecting incidental visual similarity. In contrast, our disentangled representations maintain stable performance for larger neighbourhoods. This behaviour indicates that phonological structure is preserved across broader regions of the space, yielding more coherent and robust embeddings. Together, our results show that explicit phonological supervision – and crucially, disentangled representation learning – provides a principled foundation for interpretable and transferable sign language representations. Keywords: Sign Language, Machine Learning

Machine translation from signed to spoken languages : state of the art and challenges

Mathieu De Coster, Dimitar Shterionov, Mieke Van Herreweghe, Joni Dambre

In UNIVERSAL ACCESS IN THE INFORMATION SOCIETY 2024

BIBLIO

Abstract

Automatic translation from signed to spoken languages is an interdisciplinary research domain on the intersection of computer vision, machine translation (MT), and linguistics. While the domain is growing in terms of popularity-the majority of scientific papers on sign language (SL) translation have been published in the past five years-research in this domain is performed mostly by computer scientists in isolation. This article presents an extensive and cross-domain overview of the work on SL translation. We first give a high level introduction to SL linguistics and MT to illustrate the requirements of automatic SL translation. Then, we present a systematic literature review of the state of the art in the domain. Finally, we outline important challenges for future research. We find that significant advances have been made on the shoulders of spoken language MT research. However, current approaches often lack linguistic motivation or are not adapted to the different characteristics of SLs. We explore challenges related to the representation of SL data, the collection of datasets and the evaluation of SL translation models. We advocate for interdisciplinary research and for grounding future research in linguistic analysis of SLs. Furthermore, the inclusion of deaf and hearing end users of SL translation applications in use case identification, data collection, and evaluation, is of utmost importance in the creation of useful SL translation models.

Sign language recognition with transformer networks

Mathieu De Coster, Mieke Van Herreweghe, Joni Dambre

In PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION (LREC 2020) 2020

BIBLIO

Abstract

Sign languages are complex languages. Research into them is ongoing, supported by large video corpora of which only small parts are annotated. Sign language recognition can be used to speed up the annotation process of these corpora, in order to aid research into sign languages and sign language recognition. Previous research has approached sign language recognition in various ways, using feature extraction techniques or end-to-end deep learning. In this work, we apply a combination of feature extraction using OpenPose for human keypoint estimation and end-to-end feature learning with Convolutional Neural Networks. The proven multi-head attention mechanism used in transformers is applied to recognize isolated signs in the Flemish Sign Language corpus. Our proposed method significantly outperforms the previous state of the art of sign language recognition on the Flemish Sign Language corpus: we obtain an accuracy of 74.7% on a vocabulary of 100 classes. Our results will be implemented as a suggestion system for sign language corpus annotation.