Real time sign language translator using machine learning

1. INTRODUCTION

In today's interconnected world, effective communication serves as the cornerstone of human interaction, enabling individuals to exchange information swiftly and accurately. However, for the deaf and mute community, communicating with those unfamiliar with sign language poses a significant challenge. Sign language, a sophisticated system comprising gestures, body movements, and facial expressions, serves as the primary mode of communication for this community. Yet, its complexity often hinders understanding among those not proficient in it. To bridge this communication gap and foster inclusivity, we propose the development of a groundbreaking web application. This application's core concept revolves around seamlessly translating sign language into spoken or written language and vice versa, thereby enabling a two-way communication system that is accessible and inclusive for all. The web application facilitates meaningful conversations between the deaf and mute community and individuals who can hear and speak. Users can capture sign language gestures word by word using a camera or device. Upon capture, the application leverages advanced technologies such as Tensor Flow and other image processing libraries to analyze and translate these gestures into coherent sentences or phrases.

The output is presented in text format, making it accessible to a wide range of users [1]. Moreover, the system is designed to accommodate various sign languages, including Indian Sign Language, American Sign Language, and more, and can seamlessly translate them into different spoken languages such as English and Hindi. This versatility ensures that users from diverse linguistic backgrounds can benefit from the application's functionalities. One of the system's remarkable features is its ability to translate text into sign language format, thus completing the two-way communication loop. This versatile functionality enables effective interaction between those who can hear and speak and the deaf and mute community, as messages typed by users are rendered into sign language. In essence, our web application serves as a vital bridge between these two worlds, facilitating real-time, two-way communication and fostering greater integration and understanding in our society. It leverages state-of-the-art technology to break down communication barriers, making communication more accessible and inclusive for all individuals, ultimately contributing to a more integrated and understanding society.

2. RELATED WORK

Sign language translation systems have been the focus of extensive research to bridge the communication gap between sign language users and non-signing individuals. Various approaches have been explored, including the use of deep learning models, computer vision techniques, and natural language processing (NLP) algorithms [2].

Hand Gesture Recognition: Many systems utilize Media Pipe’s Hand module or similar technologies for precise hand gesture recognition. These systems rely on computer vision algorithms to analyze the motion, shape, and position of hands and fingers.

Machine Learning Models: Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) are commonly used for sign language recognition due to their robustness in handling large datasets and variations in signing styles. These models are trained on extensive datasets to improve accuracy and context understanding.

Speech-to-Text Conversion: The integration of speech recognition technologies, such as the Google Speech API, is a common approach for converting spoken language into text. This text is then mapped to corresponding sign language gestures through a database of annotated signs.

Visualization Techniques: To display sign language gestures, systems often use animated avatars or videos, enhancing the visual representation and aiding in effective communication.

Field Research and Accuracy Analysis: To understand the practical challenges and requirements of sign language learners, I visited Mukbadhir Shala in Pandharpur. This visit provided valuable insights into how our system can be optimized for educational settings. I analyzed the interactions of students and how they use sign language, which guided the design of user-friendly and effective learning modules in our application. The feedback from teachers and students highlighted the importance of real-time processing and accuracy, which are crucial for an effective sign language translation system. During my analysis, I observed that the accuracy of gesture recognition and speech-to-text conversion significantly impacts the overall user experience, underscoring the need for robust and precise algorithms.

3. METHODOLOGY

To realize the Sign Language Translation system, we have meticulously designed and implemented various functionalities to ensure seamless communication between users proficient in sign language and those using spoken language. The implementation is structured around three key modes: Sign Language to Text, Speech to Sign Language, and Learning [3].

Sign Language to Text: Users have the option to use either a webcam or upload a video, providing flexibility in input methods. The application captures video frames in real-time and processes them using Media Pipe's Hand module, ensuring precise hand gesture recognition. Based on the detected gestures, the application identifies sign language letters and displays them on the screen, facilitating instant sign language to text translation. Additionally, a real-time video display of the hand gestures is provided for enhanced user experience. For users who wish to keep a record, the application offers the option to record the video as 'output1.mp4,' enabling them to review their interactions later.

Speech to Sign Language: This mode simplifies communication by allowing users to speak into the microphone. The application utilizes the Speech Recognition library to convert spoken words into text, ensuring accurate and real-time speech-to-text conversion. Upon recognizing the spoken words, the application displays the corresponding sign language images, promoting effective communication between speech and sign language users.

Learning: The learning mode is designed to facilitate users in learning sign language, focusing on alphabets and numbers. For alphabets, the application provides images and videos for each letter of the alphabet in American Sign Language (ASL), serving as a valuable educational resource. Similarly, for numbers, the application offers videos depicting numbers 0 to 9 in ASL, enhancing the learning experience and promoting sign language proficiency. By implementing these functionalities, our Sign Language Translation system empowers users to communicate effectively across different language modalities, fostering inclusivity and accessibility in communication. The methodologies adopted ensure accuracy, real-time processing, and user-friendly interactions, making the system a valuable tool for promoting understanding and integration in diverse communication settings.

3.2 Speech to Sign Language

Speech-to-sign language translation involves multiple steps. Initially, the Google Speech API converts spoken language into text, capturing the user's speech input. This text is then processed to identify keywords, phrases, or sentences that correspond to specific sign language gestures. The system compares the extracted text with a database of sign language gestures, which contains mappings between spoken language elements and their corresponding signs. This database is typically built through extensive data collection and annotation, associating linguistic expressions with their visual representations in sign language. Using this database, the system selects the appropriate sign language gestures that match the recognized text. This matching process can involve algorithms that consider context, grammar rules, and linguistic nuances to ensure accurate translation [6]. Once the matching signs are identified, the system generates a visual output, such as animated avatars or videos, depicting the corresponding sign language gestures. These visual representations are then displayed to the user, providing a real-time translation of their spoken input into sign language. Overall, speech-to-sign language translation systems leverage speech recognition, database matching, and visualization techniques to facilitate communication between hearing individuals and those who use sign language, enhancing accessibility and inclusivity in communication platforms.

4. RESULTS AND DISCUSSION

The user's web application for sign language translation and communication is groundbreaking, inclusive, user-friendly, and promotes real-time communication, two-way communication, and education. It is adaptable, uses advanced technologies, is affordable, and empowers individuals [8]. The application breaks down communication barriers, fosters understanding, and integrates into society. Its practicality, feasibility, and potential for future enhancements position it as an innovative and impactful tool. In the future, the application holds promise for seamless integration into video calling platforms, offering realtime sign language interpretation and expanding its sign language dataset for broader user accommodation.

Figure 3 Webcam/Video Upload: Users can choose between webcam and video upload for input, ensuring flexibility in accessing the system and capturing sign language gestures for translation.

By utilizing Media Pipe's robust hand tracking technology, the system can effectively interpret complex hand movements and gestures, ensuring accurate translation into text or spoken language. This feature significantly enhances communication accessibility for users, especially deaf and mute individuals, by providing a seamless and reliable method of translating sign language into comprehensible formats.

The integration of Media Pipe's Hand module underscores the system's commitment to leveraging cutting-edge technologies to bridge communication gaps and empower individuals with diverse communication needs [9].

Hand Gesture Recognition is a pivotal component of the system, leveraging MediaPipe's Hand module for real-time and accurate identification of hand gestures in sign language. This module employs advanced computer vision algorithms to analyze the movement, shape, and position of hands, enabling precise recognition of sign language gestures.

Figure 4 Text /audio to Image format:

It represents the conversion of text/audio into sign language format using avatars. Internally, this involves processing the input text/audio through algorithms that generate corresponding sign language gestures. These gestures are then mapped to predefined animations of avatars, creating a visual representation of sign language communication. It ensure converting into avatar/img format.

Figure 5 Alphabet/Number Learning: Users can select 'alphabets' or 'numbers' to access ASL images/videos for learning each letter and number, enhancing their sign language proficiency [10].

Users can easily navigate to the 'Alphabet' or 'Number' learning sections, where they can explore a comprehensive collection of ASL images and videos designed to facilitate effective learning of each letter and number. This interactive feature not only enhances users' sign language proficiency but also promotes a user-friendly and engaging learning experience within the platform.

Model Making Experiments: Conducted experiments to produce models with high accuracy by varying parameters.

Effect on the Number of Convolution Layers: Increasing epochs did not significantly improve accuracy but increased processing time.

Effect Optimizer: Adam optimizer yielded the best accuracy of 86%, outperforming SGD and RMSprop. Effect of Learning Rate: A learning rate of 0.001 resulted in the highest accuracy of 86%, indicating better processing with a smaller learning rate.

5. CONCLUSION

The project successfully developed a groundbreaking sign language translation system, achieving high accuracy in recognizing hand gestures and converting them to text. This innovation promotes inclusivity and effective communication for deaf and mute individuals, breaking down communication barriers and fostering social integration. With real-time translation capabilities and a user-friendly interface, the system empowers users to engage in two-way communication seamlessly. Its impact extends beyond technology, contributing to a more inclusive society where everyone, regardless of communication differences, can interact and participate actively, promoting understanding and inclusivity on a broader scale.

Acknowledgments

The authors have no specific acknowledgments to make for this research.

Funding Information

This research received no specific grant from any funding agency in the public, commercial, or notfor-profit sectors.

Author Contributions Statement

C: Conceptualization; M: Methodology; So: Software; Va: Validation; Fo: Formal analysis; I: Investigation; R: Resources; D: Data curation; O: Writing – original draft; E: Writing – review & editing; Vi: Visualization; Su: Supervision; P: Project administration; Fu: Funding acquisition.

Conflict of Interest Statement

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Informed Consent

All participants were informed about the purpose of the study, and their voluntary consent was obtained prior to data collection.

Ethical Approval

The study was conducted in compliance with the ethical principles outlined in the Declaration of Helsinki and approved by the relevant institutional authorities.

Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

1. [1] Spread The Sign - Sign Language Dictionary. www.spreadthesign.com/
2. [2] Lifeprint. "Lifeprint - American Sign Language (ASL) Learning Resources and Lessons." Lifeprint, www.lifeprint.com/. Signing Savvy. "Signing Savvy - ASL Sign Language Video Dictionary." Signing savvy, www.signingsavvy.com/.
3. [3] SignSchool. "SignSchool - Learn Sign Language Online."
4. [4] SignSchool, www.signschool.com/. ASL Deafined. "ASL Deafined - American Sign Language Learning Platform." ASL Deafined, www.asldeafined.com/.
5. [5] Signly. "Signly - Sign Language Translation Mobile App." Signly, www.signly.co/. Gallaudet University Press. "
6. [6] Gallaudet University Press - Publisher of Books on Deaf Studies and Sign Language." Gallaudet University Press, www.gallaudet.edu/gupress.
7. [7] Handspeak. "Handspeak - American Sign Language (ASL) Dictionary and Learning Resources." Handspeak, www.handspeak.com/.
8. [8] Sign Language 101 - Online ASL Course Offers a structured online course for learning ASL, suitable for self-paced learning. https://www.signlanguage101.com/
9. [9] The ASL App - Learn Sign Language Mobile app designed to help users learn ASL through video lessons and interactive features. https://theaslapp.com/
10. [10] ASLPro.com - Free American Sign Language Video Dictionary and Resources an online video dictionary with resources for teachers and learners of ASL. http://www.aslpro.com/