NVIDIA has introduced Signs, an AI-powered platform designed to improve American Sign Language (ASL) learning and accessibility.
Developed in partnership with the American Society for Deaf Children (ASDC) and digital agency Hello Monday, the system trains artificial intelligence models by incorporating real-world ASL gestures from users. The project aims to refine AI-based sign recognition by making it more accurate, scalable, and capable of understanding natural signing styles.
The initiative builds on an increasing number of AI-driven sign language projects, including SignLLM, a model designed to generate skeletal pose representations for eight different sign languages.
Unlike SignLLM, which primarily focuses on text-based translations into sign gestures, NVIDIA’s approach relies on validated video data to refine AI recognition, making it more adaptable to real-world ASL use.
How NVIDIA’s Signs Platform Works
Unlike static sign language dictionaries, Signs is designed to learn and improve through user contributions. Fluent ASL signers and learners can submit video recordings of signed words, which are reviewed by ASL linguists before being incorporated into the dataset.
NVIDIA’s goal is to build a training model that understands ASL as it is naturally used, rather than relying on rigid, predefined movements.
The dataset currently consists of 400,000 video clips covering 1,000 signed words. However, NVIDIA plans to significantly expand this collection by opening the platform to a larger audience, allowing more contributions from signers worldwide.
The company has also confirmed that it will release portions of this dataset publicly to support AI researchers working on accessibility-focused applications.
According to Cheri Dowling, executive director of ASDC, early ASL exposure plays a crucial role in language development for deaf children. She explains: “Most deaf children are born to hearing parents. Giving family members accessible tools like Signs to start learning ASL early enables them to open an effective communication channel with children as young as six to eight months old.”
AI-Powered Real-Time Feedback and Gesture Analysis
One of the key features of Signs is its ability to provide real-time AI feedback. Using a webcam, users can sign a word, and the system will analyze their gestures, offering corrections where necessary. A 3D avatar demonstrates the proper ASL sign, allowing users to compare their movements with validated examples.
Real-time gesture analysis not only enhances ASL learning but also provides NVIDIA with data to improve AI’s ability to recognize natural variations in signing. The system continuously refines its recognition model by incorporating user-submitted examples, making it more responsive to different signing speeds, styles, and hand positions.
How NVIDIA’s Approach Differs From Other AI Sign Language Models
NVIDIA’s Signs enters a space where multiple companies have been experimenting with AI-driven sign language recognition. In 2019, Google introduced AI-powered hand-tracking technology to detect gestures, while Meta has explored sign language translation using computer vision models.
Microsoft has been integrating AI into accessibility tools, including real-time captions and sign recognition for video conferencing. Microsoft and OpenAI are also collaborating with Be My Eyes, a company that provides live video assistance to people who are blind or visually impaired, aiming to make AI more accessible for these users.
What sets NVIDIA’s platform apart is its approach to real-world learning. Rather than relying on pre-programmed sign movements or simulated gestures, Signs learns from real user-submitted videos that are reviewed by ASL experts. This validation process ensures that AI isn’t just recognizing signs in isolation but understanding how they are used in practice.
Challenges in AI Sign Language Recognition
Training artificial intelligence to understand sign language presents unique challenges that extend beyond traditional speech or text recognition. ASL, like other sign languages, conveys meaning through a combination of hand movements, facial expressions, and spatial positioning.
Many AI models struggle with this complexity, as most rely primarily on hand-tracking and overlook the nuances of non-manual signals such as eyebrow movements or head tilts, which can change the meaning of a sign.
NVIDIA’s dataset validation process attempts to address this by ensuring AI learns from authentic, real-world ASL use. However, achieving high accuracy remains difficult due to regional and dialectical variations in ASL.
Two people signing the same word might use different motions depending on their background, making it harder for AI to generalize recognition patterns. While SignLLM takes a multilingual approach by generating skeletal poses for different sign languages, NVIDIA’s focus is currently on refining ASL before potentially expanding into other languages.
Another limitation of AI-based sign recognition is the reliance on large, diverse datasets. Historically, most AI sign language projects have used smaller, controlled datasets that do not accurately reflect how sign language is used in everyday conversations.
This has contributed to AI models that perform well in structured environments but struggle in real-world scenarios where lighting, camera angles, and signing styles vary widely. NVIDIA’s approach, which actively incorporates new user-submitted examples into its training data, is an attempt to bridge this gap.
Potential Applications for AI-Powered ASL Recognition
As AI becomes more proficient in recognizing sign language, its potential applications extend beyond education. Video conferencing platforms could integrate real-time AI-generated ASL captions, allowing Deaf and Hard-of-Hearing participants to engage more fully in meetings without relying on human interpreters.
Similarly, AI-driven AR glasses could provide instant sign language translation in real-world interactions, improving accessibility in public spaces.
While AI is still far from matching human interpreters in fluency, NVIDIA’s dataset-focused approach signals a shift in how artificial intelligence learns sign language. Rather than treating ASL as a set of predefined gestures, AI models are increasingly being trained to recognize the fluid, context-dependent nature of signing.
This opens up possibilities for AI-driven tutoring, real-time ASL translation, and even AI-generated sign language avatars.
One major area of interest is whether NVIDIA will eventually expand its dataset to include non-manual ASL components. Some researchers have already explored the role of facial expressions and lip movements in sign recognition, but most AI models remain limited in this regard.
NVIDIA’s work with organizations such as the Rochester Institute of Technology suggests that broader AI learning techniques could be applied to ASL recognition in the future.
The future of AI-driven ASL tools will depend on how well these models adapt to real-world usage. With platforms like Signs, NVIDIA is attempting to move beyond static training datasets and create a learning system that evolves over time. If successful, this could lay the foundation for AI-powered accessibility tools that function as dynamic learning systems rather than rigid translation engines.
