SSC is a device that can read the signs and gestures of a person and convert it to simple speech. Mute people are usually acquainted with sign language and that acts as input to the system.
The signs, based on finger movements and angle of the hands are decoded to specific words using a Deep Learning model deployed on a microcontroller, which is further fed to a speaker. Custom dataset was created for the same and the entire system is compact, portable, light-weight and power efficient.
People who are mute or have difficulties in talking rely on sign language and gestures to communicate. This becomes really difficult when the communication happens between a mute person and a normal person, who is not accustomed to sign language. The person has to therefore use simpler gestures or write down to communicate which makes it tedious, slow and is not suitable in different situations. This issue is not only applicable for people who lost their voice, but also people who undergo surgeries in their throat and cannot talk for prolonged periods of time.
We wanted to focus on this aspect and developed a mobile system that can actually help mute people in conveying their thoughts to the outside world, who don’t know or are not accustomed to the sign language system.
Our solution to this involves a sign language to speech converter device that can read the signs and gestures of a person and convert it to simple speech. Mute people are usually acquainted with sign language and that acts as input to the system. The signs, based on finger movements and angle of the hands, can be decoded to specific words which are then fed to a speaker with a custom voice set. Hence it removes the barrier of proper communication between such people and normally enabled humans. The system will also be tuned and calibrated to a specific person according to hand and finger movements to improve speech, efficiency and accuracy.
Our device uses signs and gestures as input. We use flex sensors and inertial measurement units (IMU) to properly obtain the input signals. These input signals are used to decode the word/content for communication. For tracking gestures, the IMU data will be used to predict the angle of the hand. The data/content predicted is then fed to audio output based on pre-set voice. The entire system is battery powered making it mobile. The main aim is to process everything quickly and make maximum predictions possible compared to contemporary designs and devices.
The proposed solution involves a pair of lightweight gloves equipped with nonintrusive sensors which enhances user experience and convenience. This solution avoids use of expensive high-resolution cameras which are used in traditional visionbased gesture recognition systems. The hand gesture recognition model is run on a microcontroller, whereas traditional solutions involve much powerful and expensive microprocessor-based systems like Raspberry Pi. Use of a microcontroller in place of a microprocessor-based system is advantageous since it has reduced power consumption, occupies less space and is comparatively lighter.
