In sign languages facial expressions and body language function as core grammatical tools known as non-manual markers, used in grammar and syntax, distinguishing adverbs and adjectives, indicating tone and intensity. Classifiers, abbreviated as CL, are signed morphemes (particularly, classifier handshapes) that represent a group of nouns or referents.
Fingerspelling hand shapes are the building blocks for the hand-related elements of the language (signs done with the hands) whose meaning is dependent on the context, much like a spoken language's word's meaning can be affected by where in a sentence it is placed, tone/pitch, etc.
From a technical viewpoint, if you intend to push beyond where most projects similar to this one have ended up, it might be interesting to consider SL handshapes as analogous to tokens in an LLM, vector encoding them to capture the context and semantic meaning.
From the human relationship POV, making an effort to learn full natural sign language to communicate with your relatives will no doubt make them very happy and enrich their lives and yours.
loicalleyne
I know this is well meaning, but you should know that in the disability community it's very common to have people invent tech for us without fully understanding our needs. (Smart canes, for instance.)
What your demo shows is a model understanding fingerspelling. Fingerspelling is ... a very small part of sign language.
To make the claim you are translating sign language you need to understand facial expressions, hand positioning/motion (help you vs help me, same hand shape, different positions/motions), classifiers, and a ton more stuff.
It's a really beautiful language, and I encourage everyone to learn it, but for your project I'd be very careful to not make a claim about translating sign language.
In the mean time, it sounds like you've got some passion for accessibility work? We're always looking for more skilled engineers to work in the a11y space. It does require some humility and willingness to listen to others needs.
forestrywat
As others have pointed out, it might be helpful to step back and learn more about sign language. More importantly though, you might want to learn more about Deaf culture and how this type of technology might be received
cush
I think small projects like this are how better technology eventually gets built. Excited to see where you take it.
ethannam9022
You know, it might be time for me to take up woodworking...
comments (6)
In sign languages facial expressions and body language function as core grammatical tools known as non-manual markers, used in grammar and syntax, distinguishing adverbs and adjectives, indicating tone and intensity. Classifiers, abbreviated as CL, are signed morphemes (particularly, classifier handshapes) that represent a group of nouns or referents. Fingerspelling hand shapes are the building blocks for the hand-related elements of the language (signs done with the hands) whose meaning is dependent on the context, much like a spoken language's word's meaning can be affected by where in a sentence it is placed, tone/pitch, etc.
From a technical viewpoint, if you intend to push beyond where most projects similar to this one have ended up, it might be interesting to consider SL handshapes as analogous to tokens in an LLM, vector encoding them to capture the context and semantic meaning.
From the human relationship POV, making an effort to learn full natural sign language to communicate with your relatives will no doubt make them very happy and enrich their lives and yours.
loicalleyne
What your demo shows is a model understanding fingerspelling. Fingerspelling is ... a very small part of sign language.
To make the claim you are translating sign language you need to understand facial expressions, hand positioning/motion (help you vs help me, same hand shape, different positions/motions), classifiers, and a ton more stuff.
It's a really beautiful language, and I encourage everyone to learn it, but for your project I'd be very careful to not make a claim about translating sign language.
In the mean time, it sounds like you've got some passion for accessibility work? We're always looking for more skilled engineers to work in the a11y space. It does require some humility and willingness to listen to others needs.
forestrywat
cush
ethannam9022
squigz
bradleykingz