Sign Language Computational Linguistics

About this research line

We research sign language processing in collaboration with (among others) the Flemish Sign Language community, to enable co-created AI-driven sign language technology.

In Flanders, only about 13,000 people can communicate in Flemish sign language (Vlaamse Gebarentaal, VGT). For many of those people, VGT is their preferred language.

Since most hearing people do not understand sign language, signers and non-signers mostly communicate through interpreters, or through written language. Neither is practical for day-to-day iteraction, or getting to know each other on an informal basis. Interpreters are only available by appointment and need to be paid, and not all signers are equally fluent in written communication.

If each person could communicate in the language they feel most familiar with, communication could become a lot easier. In the European SignON project, we leveraged machine learning and AI to automatically translate between different European sign languages and different spoken or written languages. While automatically translating from or into sign languages for open, informal communication is still a very far-off goal, we believe that a first step in the form of communication in specific use cases or scenarios is feasible within the next few years.

The whole SignON platform development is user-driven, with a strong participation of native signers and the deaf communities from different countries. For the technical development, IDLab-AIRO investigated the use of deep learning techniques in order to create a sign language recognition and understanding system. Other partners contributed to the translation and language generation parts.

Sign languages are complex visual languages, with some specific properties that are not present in spoken or written languages. In order to understand these properties and incorporate them in our model development, several experts in sign language linguistics are also involved in the project.

From a deep learning perspective, another difficulty is the fact that only very small labeled datasets are available, at least in comparison to those for speech recognition and natural language processing on written text. Furthermore, sign languages have their own grammar and dialects. This makes sign language recognition and translation a very challenging and very exciting problem from the perspective of data efficiency.

At IDLab-AIRO, our current research builds on the pioneering work of one of our alumni, dr. Lionel Pigou. His research into the use of convolutional neural networks for sign language recognition is still highly cited in the domain today.

Our goal is to use domain and task knowledge to increase the performance of sign language recognition models to the point of usability.

Active researchers

Related publications

Signbuddy : from sign language research to scalable co-created solutions

Toon Vandendriessche, Caro Brosens, Hannes De Durpel, Mathieu De Coster, Joni Dambre

In UNIVERSAL ACCESS IN THE INFORMATION SOCIETY 2026

BIBLIO

Abstract

This paper presents SignBuddy, the result of ongoing co-created sign language processing research. Mostsign language processing research is performed by hearing, non-signing researchers. Even though co-creation efforts have recently increased, technical research still often fails to mention if (and how) co-creation was involved in the research process. SignBuddy is a co-created research tool developed through apartnership between the Flemish Sign Language Centre, a deaf-led organisation, and Ghent University. While respecting elemental concepts of co-creation - i.e. (i) defining common goals and (ii) building a formal and sustainable relationship between users/consumers and researchers/developers and respectingthe five lessons in co-creation - the platform successfully supported the development of the first fully scalable sign-to-text dictionary search system, built into the Flemish Sign Language-Dutch onlinedictionary. SignBuddy functions as a crowdsourcing interface for in-the-wild collection of modelevaluation data, gathering example queries for quantitative performance analysis and user feedback forqualitative assessment. This human evaluation allows us to shape the application based on the end-users'needs. Addressing the need for models that support large dictionaries (over ten thousand signs), we propose a scalable one-shot sign language recognition method and achieve state-of-the-art results. Beyond the co-created application itself, this work provides insights into the co-creation process - clarifying roles, shared goals, and responsibilities - and offers conclusions to guide future co-created sign language processing research.

Machine translation from signed to spoken languages : state of the art and challenges

Mathieu De Coster, Dimitar Shterionov, Mieke Van Herreweghe, Joni Dambre

In UNIVERSAL ACCESS IN THE INFORMATION SOCIETY 2024

BIBLIO

Abstract

Automatic translation from signed to spoken languages is an interdisciplinary research domain on the intersection of computer vision, machine translation (MT), and linguistics. While the domain is growing in terms of popularity-the majority of scientific papers on sign language (SL) translation have been published in the past five years-research in this domain is performed mostly by computer scientists in isolation. This article presents an extensive and cross-domain overview of the work on SL translation. We first give a high level introduction to SL linguistics and MT to illustrate the requirements of automatic SL translation. Then, we present a systematic literature review of the state of the art in the domain. Finally, we outline important challenges for future research. We find that significant advances have been made on the shoulders of spoken language MT research. However, current approaches often lack linguistic motivation or are not adapted to the different characteristics of SLs. We explore challenges related to the representation of SL data, the collection of datasets and the evaluation of SL translation models. We advocate for interdisciplinary research and for grounding future research in linguistic analysis of SLs. Furthermore, the inclusion of deaf and hearing end users of SL translation applications in use case identification, data collection, and evaluation, is of utmost importance in the creation of useful SL translation models.

Sign language recognition with transformer networks

Mathieu De Coster, Mieke Van Herreweghe, Joni Dambre

In PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION (LREC 2020) 2020

BIBLIO

Abstract

Sign languages are complex languages. Research into them is ongoing, supported by large video corpora of which only small parts are annotated. Sign language recognition can be used to speed up the annotation process of these corpora, in order to aid research into sign languages and sign language recognition. Previous research has approached sign language recognition in various ways, using feature extraction techniques or end-to-end deep learning. In this work, we apply a combination of feature extraction using OpenPose for human keypoint estimation and end-to-end feature learning with Convolutional Neural Networks. The proven multi-head attention mechanism used in transformers is applied to recognize isolated signs in the Flemish Sign Language corpus. Our proposed method significantly outperforms the previous state of the art of sign language recognition on the Flemish Sign Language corpus: we obtain an accuracy of 74.7% on a vocabulary of 100 classes. Our results will be implemented as a suggestion system for sign language corpus annotation.