Sign Language Computational Linguistics

About this research line

We research sign language processing in collaboration with (among others) the Flemish Sign Language community, to co-create AI-driven sign language technology.

In Flanders, only about 13,000 people can communicate in Flemish Sign Language (Vlaamse Gebarentaal, VGT). For many of those people, VGT is their preferred language. Since most hearing people do not understand sign language, signers and non-signers mostly communicate through interpreters or through written language. Neither is practical for ad-hoc or day-to-day interaction, or for getting to know each other on an informal basis. Interpreters are only available by appointment and need to be paid, and not all signers are equally fluent in written communication.

In recent years, AI-powered tools like ChatGPT, Gemini and Whisper have revolutionised spoken language communication. However, sign language technology lags behind for several reasons. Data scarcity, the limited level of standardisation, the fragmentation of sign language communities, the often highly improvisational nature of sign languages and the relatively small user base all contribute to this. At IDLab AIRO, we directly address some of these factors that hinder sign language technology development.

Our recent results build on the pioneering work of our alumni, Dr Lionel Pigou and Dr Mathieu De Coster, whose research focused on automatically recognising individual signs. This culminated in the co-creation, with the Flemish Sign Language Centre (VGTC), of SignBuddy — an AI-powered data collection tool for isolated signs ‘in the wild’. The collected data was used to validate and refine a scalable, large dictionary search method, which was later released into the Flemish Sign Language Dictionary (https://woordenboek.vlaamsegebarentaal.be/) as the world’s first publicly available sign-to-text dictionary search. Users of the VGT dictionary can now search through 12,000 signs for translations simply by performing any sign. Both SignBuddy and the sign-to-text search in the VGT dictionary are successful outcomes of our continuous co-creation process with VGTC. This collaboration ensures community requirements and preferences are safeguarded throughout.

More recently, our research focus has shifted from proficient sign recognition to sign language computational linguistics, and more specifically to understanding the independent sign parameters – phonemes – that constitute sign language, and their representation in a sign language technology context. However, the same vision still drives our research: producing useful and meaningful outputs for the Deaf and Hard of Hearing communities in the short term, which can serve as stepping stones for larger developments in the long term. This fosters community-driven research and preserves the co-creation relationships that are fundamental to our approach.

Active researchers

Related publications

Signbuddy : from sign language research to scalable co-created solutions

Toon Vandendriessche, Caro Brosens, Hannes De Durpel, Mathieu De Coster, Joni Dambre

In UNIVERSAL ACCESS IN THE INFORMATION SOCIETY 2026

BIBLIO

Abstract

This paper presents SignBuddy, the result of ongoing co-created sign language processing research. Mostsign language processing research is performed by hearing, non-signing researchers. Even though co-creation efforts have recently increased, technical research still often fails to mention if (and how) co-creation was involved in the research process. SignBuddy is a co-created research tool developed through apartnership between the Flemish Sign Language Centre, a deaf-led organisation, and Ghent University. While respecting elemental concepts of co-creation - i.e. (i) defining common goals and (ii) building a formal and sustainable relationship between users/consumers and researchers/developers and respectingthe five lessons in co-creation - the platform successfully supported the development of the first fully scalable sign-to-text dictionary search system, built into the Flemish Sign Language-Dutch onlinedictionary. SignBuddy functions as a crowdsourcing interface for in-the-wild collection of modelevaluation data, gathering example queries for quantitative performance analysis and user feedback forqualitative assessment. This human evaluation allows us to shape the application based on the end-users'needs. Addressing the need for models that support large dictionaries (over ten thousand signs), we propose a scalable one-shot sign language recognition method and achieve state-of-the-art results. Beyond the co-created application itself, this work provides insights into the co-creation process - clarifying roles, shared goals, and responsibilities - and offers conclusions to guide future co-created sign language processing research.

Grounding sign language representation learning in phonology

Toon Vandendriessche, Mathieu De Coster, Joni Dambre

In Proceedings 12th Workshop on the Representation and Processing of Sign Languages : Language in Motion 2026

BIBLIO

Abstract

Sign language recognition systems are commonly trained using gloss-level supervision, treating signs as holistic lexical units. While effective for classification, such approaches entangle sub-lexical structure and fail to capture the phonological parameters that govern sign formation, limiting interpretability, robustness, and cross-lingual transfer. In this work, we propose a phonologically informed representation learning architecture that explicitly structures the latent space according to linguistic principles. Grounded in the Dependency Model – a phonological model used to describe Flemish Sign Language (VGT) – our hierarchical architecture disentangles parameter-specific subspaces for handshape and location and is trained with multi-label phoneme supervision. To evaluate whether phonological information is directly encoded in the geometry of the embedding space, we introduce a non-parametric probing method that measures neighbourhood consistency across increasing scales. We show that conventional gloss-based networks achieve reasonable performance only for very small neighbourhoods, reflecting incidental visual similarity. In contrast, our disentangled representations maintain stable performance for larger neighbourhoods. This behaviour indicates that phonological structure is preserved across broader regions of the space, yielding more coherent and robust embeddings. Together, our results show that explicit phonological supervision – and crucially, disentangled representation learning – provides a principled foundation for interpretable and transferable sign language representations. Keywords: Sign Language, Machine Learning

Sign language recognition with transformer networks

Mathieu De Coster, Mieke Van Herreweghe, Joni Dambre

In PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION (LREC 2020) 2020

BIBLIO

Abstract

Sign languages are complex languages. Research into them is ongoing, supported by large video corpora of which only small parts are annotated. Sign language recognition can be used to speed up the annotation process of these corpora, in order to aid research into sign languages and sign language recognition. Previous research has approached sign language recognition in various ways, using feature extraction techniques or end-to-end deep learning. In this work, we apply a combination of feature extraction using OpenPose for human keypoint estimation and end-to-end feature learning with Convolutional Neural Networks. The proven multi-head attention mechanism used in transformers is applied to recognize isolated signs in the Flemish Sign Language corpus. Our proposed method significantly outperforms the previous state of the art of sign language recognition on the Flemish Sign Language corpus: we obtain an accuracy of 74.7% on a vocabulary of 100 classes. Our results will be implemented as a suggestion system for sign language corpus annotation.