Sign Language Computational Linguistics

About this research line

We research sign language processing in collaboration with (among others) the Flemish Sign Language community, to co-create AI-driven sign language technology.

In Flanders, only about 13,000 people can communicate in Flemish Sign Language (Vlaamse Gebarentaal, VGT). For many of those people, VGT is their preferred language. Since most hearing people do not understand sign language, signers and non-signers mostly communicate through interpreters or through written language. Neither is practical for ad-hoc or day-to-day interaction, or for getting to know each other on an informal basis. Interpreters are only available by appointment and need to be paid, and not all signers are equally fluent in written communication.

In recent years, AI-powered tools like ChatGPT, Gemini and Whisper have revolutionised spoken language communication. However, sign language technology lags behind for several reasons. Data scarcity, the limited level of standardisation, the fragmentation of sign language communities, the often highly improvisational nature of sign languages and the relatively small user base all contribute to this. At IDLab AIRO, we directly address some of these factors that hinder sign language technology development.

Our recent results build on the pioneering work of our alumni, Dr Lionel Pigou and Dr Mathieu De Coster, whose research focused on automatically recognising individual signs. This culminated in the co-creation, with the Flemish Sign Language Centre (VGTC), of SignBuddy — an AI-powered data collection tool for isolated signs ‘in the wild’. The collected data was used to validate and refine a scalable, large dictionary search method, which was later released into the Flemish Sign Language Dictionary (https://woordenboek.vlaamsegebarentaal.be/) as the world’s first publicly available sign-to-text dictionary search. Users of the VGT dictionary can now search through 12,000 signs for translations simply by performing any sign. Both SignBuddy and the sign-to-text search in the VGT dictionary are successful outcomes of our continuous co-creation process with VGTC. This collaboration ensures community requirements and preferences are safeguarded throughout.

More recently, our research focus has shifted from proficient sign recognition to sign language computational linguistics, and more specifically to understanding the independent sign parameters – phonemes – that constitute sign language, and their representation in a sign language technology context. However, the same vision still drives our research: producing useful and meaningful outputs for the Deaf and Hard of Hearing communities in the short term, which can serve as stepping stones for larger developments in the long term. This fosters community-driven research and preserves the co-creation relationships that are fundamental to our approach.

Active researchers

Related publications

Signbuddy : from sign language research to scalable co-created solutions

Toon Vandendriessche, Caro Brosens, Hannes De Durpel, Mathieu De Coster, Joni Dambre

In UNIVERSAL ACCESS IN THE INFORMATION SOCIETY 2026

BIBLIO

Abstract

This paper presents SignBuddy, the result of ongoing co-created sign language processing research. Mostsign language processing research is performed by hearing, non-signing researchers. Even though co-creation efforts have recently increased, technical research still often fails to mention if (and how) co-creation was involved in the research process. SignBuddy is a co-created research tool developed through apartnership between the Flemish Sign Language Centre, a deaf-led organisation, and Ghent University. While respecting elemental concepts of co-creation - i.e. (i) defining common goals and (ii) building a formal and sustainable relationship between users/consumers and researchers/developers and respectingthe five lessons in co-creation - the platform successfully supported the development of the first fully scalable sign-to-text dictionary search system, built into the Flemish Sign Language-Dutch onlinedictionary. SignBuddy functions as a crowdsourcing interface for in-the-wild collection of modelevaluation data, gathering example queries for quantitative performance analysis and user feedback forqualitative assessment. This human evaluation allows us to shape the application based on the end-users'needs. Addressing the need for models that support large dictionaries (over ten thousand signs), we propose a scalable one-shot sign language recognition method and achieve state-of-the-art results. Beyond the co-created application itself, this work provides insights into the co-creation process - clarifying roles, shared goals, and responsibilities - and offers conclusions to guide future co-created sign language processing research.

Machine translation from signed to spoken languages : state of the art and challenges

Mathieu De Coster, Dimitar Shterionov, Mieke Van Herreweghe, Joni Dambre

In UNIVERSAL ACCESS IN THE INFORMATION SOCIETY 2024

BIBLIO

Abstract

Automatic translation from signed to spoken languages is an interdisciplinary research domain on the intersection of computer vision, machine translation (MT), and linguistics. While the domain is growing in terms of popularity-the majority of scientific papers on sign language (SL) translation have been published in the past five years-research in this domain is performed mostly by computer scientists in isolation. This article presents an extensive and cross-domain overview of the work on SL translation. We first give a high level introduction to SL linguistics and MT to illustrate the requirements of automatic SL translation. Then, we present a systematic literature review of the state of the art in the domain. Finally, we outline important challenges for future research. We find that significant advances have been made on the shoulders of spoken language MT research. However, current approaches often lack linguistic motivation or are not adapted to the different characteristics of SLs. We explore challenges related to the representation of SL data, the collection of datasets and the evaluation of SL translation models. We advocate for interdisciplinary research and for grounding future research in linguistic analysis of SLs. Furthermore, the inclusion of deaf and hearing end users of SL translation applications in use case identification, data collection, and evaluation, is of utmost importance in the creation of useful SL translation models.

Sign language recognition with transformer networks

Mathieu De Coster, Mieke Van Herreweghe, Joni Dambre

In PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION (LREC 2020) 2020

BIBLIO

Abstract

Sign languages are complex languages. Research into them is ongoing, supported by large video corpora of which only small parts are annotated. Sign language recognition can be used to speed up the annotation process of these corpora, in order to aid research into sign languages and sign language recognition. Previous research has approached sign language recognition in various ways, using feature extraction techniques or end-to-end deep learning. In this work, we apply a combination of feature extraction using OpenPose for human keypoint estimation and end-to-end feature learning with Convolutional Neural Networks. The proven multi-head attention mechanism used in transformers is applied to recognize isolated signs in the Flemish Sign Language corpus. Our proposed method significantly outperforms the previous state of the art of sign language recognition on the Flemish Sign Language corpus: we obtain an accuracy of 74.7% on a vocabulary of 100 classes. Our results will be implemented as a suggestion system for sign language corpus annotation.