-
-
Notifications
You must be signed in to change notification settings - Fork 87
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(landing): add numbers section backed up by sources from docs
- Loading branch information
Showing
13 changed files
with
309 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
# Awards and Recognition | ||
|
||
In this page, we list the awards and recognition that the technology behind sign.mt has received in the last few years, alongside public independent reviews. | ||
|
||
## 2024 | ||
|
||
- We received an `EMNLP Outstanding Demo Paper Award` for our demonstration of "sign.mt: Real-Time Multilingual Sign Language Translation Application"[^emnlp2024-award]. | ||
|
||
[^emnlp2024-award]: sign.mt. 2024. [Twitter Post](https://x.com/signmt_/status/1857181686045540787). | ||
|
||
- Dr. Amit Moryossef (among others) received the `SwissNLP Award` for his "outstanding contribution to Sign Language Translation in Switzerland"[^swissnlp2024-award]. | ||
|
||
[^swissnlp2024-award]: SwissNLP. 2024. [SwissNLP Award](https://swissnlp.org/home/activities/swissnlp-award/). | ||
|
||
- sign.mt was independently reviewed for Kazakh-Russian Sign Language translation, reporting accuracy of 37% compared to 98% for human translation[^kazakh-russian-review]. | ||
|
||
[^kazakh-russian-review]: Imashev et al. 2024. [Comparative Analysis of Sign Language Interpreting Agents Perception: A Study of the Deaf](hhttps://aclanthology.org/2024.lrec-main.319/). | ||
|
||
## 2023 | ||
|
||
- Dr. Amit Moryossef (among others) received the `ACL Outstanding Paper Award` for "Considerations for Meaningful Sign Language Machine Translation"[^acl2023-award]. | ||
|
||
[^acl2023-award]: Müller et al. 2023. [Considerations for Meaningful Sign Language Machine Translation](https://aclanthology.org/2023.acl-short.60/). | ||
|
||
## 2021 | ||
|
||
- Dr. Amit Moryossef (among others) received the `ACL Best Theme Paper Award` for "Including Signed Languages in Natural Language Processing"[^acl2021-award]. | ||
|
||
[^acl2021-award]: Yin et al. 2021. [Including Signed Languages in Natural Language Processing](https://aclanthology.org/2021.acl-long.570/). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,126 @@ | ||
# Our Approach | ||
|
||
Following the research of Dr. Amit Moryosef's published in his PhD thesis[^amit-thesis], we aim to develop a sign language translation system that separates the computer vision tasks from the language translation tasks. | ||
|
||
[^amit-thesis]: Amit Moryosef. 2024. [Real-Time Multilingual Sign Language Processing](https://arxiv.org/abs/2412.01991). | ||
|
||
## Spoken to Signed Language Translation | ||
|
||
Following, is a flowchart of the current translation pipeline from spoken to signed language. | ||
Each node represents a different module or function in the pipeline, with a link to the relevant code repository. | ||
|
||
- <span style="color: green">Green</span> edges represent high quality modules. | ||
- <span style="color: orange">orange</span> edges represent existing, low quality modules. | ||
- <span style="color: red">red</span> edges represent modules that need to be implemented. | ||
|
||
```mermaid | ||
flowchart TD | ||
A0[Spoken Language Audio] --> A1(Spoken Language Text) | ||
A1[Spoken Language Text] --> B[<a href='https://github.com/sign/translate/issues/10'>Language Identification</a>] | ||
A1 --> C(<a href='https://github.com/sign/translate/tree/master/functions/src/text-normalization'>Normalized Text</a>) | ||
B --> C | ||
C & B --> Q(<a href='https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/Segmenter'>Sentence Splitter</a>) | ||
Q & B --> D(<a href='https://github.com/sign-language-processing/signbank-plus'>SignWriting</a>) | ||
C -.-> M(<a href='https://github.com/ZurichNLP/spoken-to-signed-translation' title='We would like to move away from glosses'>Glosses</a>) | ||
M -.-> E | ||
D --> E(<a href='https://github.com/sign-language-processing/signwriting-animation'>Pose Sequence</a>) | ||
D -.-> I(<a href='https://github.com/sign-language-processing/signwriting-illustration'>Illustration</a>) | ||
N --> H(<a href='https://github.com/sign/translate/issues/68'>3D Avatar</a>) | ||
N --> G(<a href='https://github.com/sign-language-processing/pose'>Skeleton Viewer</a>) | ||
N --> F(<a href='https://github.com/sign-language-processing/pose-to-video' title='Help wanted!'>Human GAN</a>) | ||
H & G & F --> J(Video) | ||
J --> K(Share Translation) | ||
D -.-> L(<a href='https://github.com/sign-language-processing/signwriting-description' title='Poor performance. Help wanted!'>Description</a>) | ||
O --> N(<a href='https://github.com/sign-language-processing/fluent-pose-synthesis' title='Currently skipped. Help Wanted!'>Fluent Pose Sequence</a>) | ||
E --> O(<a href='https://github.com/sign-language-processing/pose-anonymization'>Pose Appearance Transfer</a>) | ||
linkStyle default stroke:green; | ||
linkStyle 3,5,7 stroke:lightgreen; | ||
linkStyle 10,11,12,15 stroke:red; | ||
linkStyle 6,8,9,14,19,20 stroke:orange; | ||
``` | ||
|
||
This pipeline in fact represents two types of approaches to translation: | ||
|
||
1. **Dictionary Based Translation** | ||
2. **SignWriting based Machine Translation** | ||
|
||
### Dictionary Based Translation[^dictionary-baseline] | ||
|
||
[^dictionary-baseline]: Moryossef et al. 2023. [An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation](https://aclanthology.org/2023.at4ssl-1.3/). | ||
|
||
The dictionary-based translation approach aims to simplify the translation but sacrifices the fluency and natural expressiveness needed for accurate sign language communication. It can be characterized by the following general steps: | ||
|
||
```mermaid | ||
flowchart LR | ||
a[Spoken Language Text] --> b[Glosses] --> c[Pose Sequence] --> d[Video] | ||
``` | ||
|
||
![Visualization of one example through the dictionary-based translation pipeline](./assets//dictionary-pipeline.png) | ||
|
||
#### **Main Translation Steps** | ||
|
||
1. **text-to-gloss translation**: The input spoken language text is transformed into glosses using techniques such as lemmatization, word reordering, and dropping unnecessary words (like articles). | ||
|
||
2. **gloss-to-pose conversion**: ‘Glosses’ are mapped to corresponding skeletal poses extracted from a sign language dictionary. | ||
|
||
3. **pose-to-video rendering**: The pose sequences are stitched together and rendered back into a synthesized human video. | ||
|
||
#### **Data Requirements** | ||
|
||
A dictionary of sign language videos for isolated letters, words, and phrases. This dictionary cannot represent the full expressiveness of sign language. | ||
|
||
#### **Potential Quality** | ||
|
||
Even if the dictionary is complete, this approach cannot fully capture the complexity of sign language. While it attempts to translate from text to a gloss-based structure and map that to movements (poses), it fails to account for the full grammatical and syntactic richness of sign language. Glosses are an incomplete representation of actual signs, and grammar in sign language differs substantially from spoken languages. | ||
|
||
### SignWriting based Machine Translation | ||
|
||
The machine translation approach aims to achieve similar translation quality to spoken language translation systems such as Google Translate, with potentially high fluency and natural expressiveness. This approach further allows for bi-directional translation, between signed and spoken languages. | ||
|
||
```mermaid | ||
flowchart LR | ||
a[Spoken Language Text] --> b[SignWriting] --> c[Pose Sequence] --> d[Video] | ||
``` | ||
|
||
![Visualization of one example through the SignWriting-based translation pipeline](./assets/sign-tube-example.png) | ||
|
||
#### **Main Translation Steps:** | ||
|
||
1. **text-to-SignWriting Translation**: Spoken language text is translated into SignWriting sequences using machine translation (it sees a lot of data, and extrapolates from it). | ||
|
||
2. **SignWriting-to-pose conversion**: The written signs are converted into a fluent pose sequence, illustrating how the signs would be physically performed by a signer. | ||
|
||
3. **pose-to-video rendering**: This pose sequence is then rendered into a human video. | ||
|
||
#### **Data Requirements:** | ||
|
||
By combining a relatively small dataset of transcribed single signs (~100k) with a relatively small dataset of segmented continuous signs, and leveraging large video/text sign language datasets, we can automatically transcribe the latter. This process will generate large synthesized datasets for both **text-to-SignWriting** and **SignWriting-to-pose** conversions. | ||
|
||
#### **Potential Quality:** | ||
|
||
The system aims to accurately represent sign language grammar and structure, allowing for a good translation of both lexical and non-lexical signs, expressions, and classifiers. Potentially, the system can be as good as a deaf human translator, given quality data. | ||
|
||
## Signed to Spoken Language Translation | ||
|
||
Following, is a flowchart of the current translation pipeline from signed to spoken language. | ||
|
||
```mermaid | ||
flowchart TD | ||
A0[Upload Sign Language Video] --> A3[Video] | ||
A1[Camera Sign Language Video] --> A3 | ||
A3 --> B(Pose Estimation) | ||
B --> C(<a href='https://github.com/sign-language-processing/segmentation'>Segmentation</a>) | ||
C & B --> D(<a href='https://github.com/sign-language-processing/transcription'>SignWriting Transcription</a>) | ||
A2[Language Selector] --> E(<a href='https://github.com/sign-language-processing/signbank-plus'>Spoken Language Text</a>) | ||
D --> E | ||
E --> F(Spoken Language Audio) | ||
E --> G(<a href='https://github.com/sign/translate/issues/19'>Share Translation</a>) | ||
C -.-> H(Sign Image) | ||
linkStyle 1,2 stroke:orange; | ||
linkStyle 4,5,6 stroke:lightgreen; | ||
linkStyle 3,5,7,10 stroke:red; | ||
linkStyle 0,8,9 stroke:green; | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
22 changes: 22 additions & 0 deletions
22
src/app/pages/landing/about/about-numbers/about-numbers.component.html
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
<div class="container"> | ||
<h2>Our Vision: To Improve the Lives of 70 Million Deaf People Worldwide</h2> | ||
<div class="cards-container"> | ||
@for (fact of facts; track fact) { | ||
<ion-card> | ||
<ion-card-header> | ||
<ion-card-title> | ||
{{ fact.title }} | ||
@if (fact.source) { | ||
<a [href]="fact.source" target="_blank"> | ||
<ion-badge [color]="fact.color">source</ion-badge> | ||
</a> | ||
} | ||
</ion-card-title> | ||
</ion-card-header> | ||
<ion-card-content> | ||
<p>{{ fact.description }}</p> | ||
</ion-card-content> | ||
</ion-card> | ||
} | ||
</div> | ||
</div> |
28 changes: 28 additions & 0 deletions
28
src/app/pages/landing/about/about-numbers/about-numbers.component.scss
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
@import '../../../../../theme/variables'; | ||
|
||
h2 { | ||
text-align: center; | ||
} | ||
|
||
.cards-container { | ||
display: grid; | ||
grid-template-columns: repeat(3, 1fr); | ||
|
||
@media #{$breakpoint-lt-md} { | ||
grid-template-columns: repeat(2, 1fr); | ||
} | ||
|
||
@media #{$breakpoint-lt-sm} { | ||
grid-template-columns: 1fr; | ||
} | ||
} | ||
|
||
ion-card-title { | ||
display: flex; | ||
align-items: center; | ||
|
||
a { | ||
flex-grow: 1; | ||
text-align: end; | ||
} | ||
} |
31 changes: 31 additions & 0 deletions
31
src/app/pages/landing/about/about-numbers/about-numbers.component.spec.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
import {ComponentFixture, TestBed} from '@angular/core/testing'; | ||
|
||
import {AboutNumbersComponent} from './about-numbers.component'; | ||
import {provideIonicAngular} from '@ionic/angular/standalone'; | ||
import {axe, toHaveNoViolations} from 'jasmine-axe'; | ||
|
||
describe('AboutNumbersComponent', () => { | ||
let component: AboutNumbersComponent; | ||
let fixture: ComponentFixture<AboutNumbersComponent>; | ||
|
||
beforeEach(async () => { | ||
await TestBed.configureTestingModule({ | ||
imports: [AboutNumbersComponent], | ||
providers: [provideIonicAngular()], | ||
}).compileComponents(); | ||
|
||
fixture = TestBed.createComponent(AboutNumbersComponent); | ||
component = fixture.componentInstance; | ||
fixture.detectChanges(); | ||
}); | ||
|
||
it('should create', () => { | ||
expect(component).toBeTruthy(); | ||
}); | ||
|
||
it('should pass accessibility test', async () => { | ||
jasmine.addMatchers(toHaveNoViolations); | ||
const a11y = await axe(fixture.nativeElement); | ||
expect(a11y).toHaveNoViolations(); | ||
}); | ||
}); |
53 changes: 53 additions & 0 deletions
53
src/app/pages/landing/about/about-numbers/about-numbers.component.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
import {Component} from '@angular/core'; | ||
import {IonBadge, IonCard, IonCardContent, IonCardHeader, IonCardTitle} from '@ionic/angular/standalone'; | ||
|
||
@Component({ | ||
selector: 'app-about-numbers', | ||
standalone: true, | ||
imports: [IonCard, IonCardHeader, IonCardTitle, IonCardContent, IonBadge], | ||
templateUrl: './about-numbers.component.html', | ||
styleUrl: './about-numbers.component.scss', | ||
}) | ||
export class AboutNumbersComponent { | ||
facts = [ | ||
{ | ||
title: '70,000,000', | ||
description: | ||
'Globally there are 70 million deaf people. By 2050, over 700 million people could have disabling hearing loss.', | ||
source: 'https://sign.mt/docs/docs/facts/population.html#deafness', | ||
color: 'primary', | ||
}, | ||
{ | ||
title: 'Literacy Challenge', | ||
description: | ||
'Spoken language literacy rates among deaf individuals are uncertain, with estimates ranging from 20% to 50%.', | ||
source: 'https://sign.mt/docs/docs/facts/literacy.html', | ||
color: 'warning', | ||
}, | ||
{ | ||
title: '$33.7B', | ||
description: | ||
'The U.S. sign language translation market is worth $11B, with a global estimate of $33.7B across 30 developed nations.', | ||
source: 'https://sign.mt/docs/docs/facts/market.html', | ||
color: 'success', | ||
}, | ||
{ | ||
title: '3x 1million+', | ||
description: 'Our technology is being reviewed by three enterprises, each with over 1 million retail customers.', | ||
}, | ||
{ | ||
title: '+200 Languages', | ||
description: | ||
'Our translation solution currently partially supports 43 of the 200+ major sign languages to some extent, with exciting plans to expand further.', | ||
source: 'https://sign.mt/docs/docs/facts/population.html#deafness', | ||
color: 'tertiary', | ||
}, | ||
{ | ||
title: 'Reviews and Awards', | ||
description: | ||
'Independently reviewed with multiple awards, praised for innovation and effectiveness in Natural Language Processing.', | ||
source: 'https://sign.mt/docs/docs/technology/awards.html', | ||
color: 'danger', | ||
}, | ||
]; | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters