Dell Latitude 5521, 11th Gen Intel Core i5-11400H @ 2.70GHz x12, RAM 16 GB, SSD SK hynix PC711 256GB.
| Tokenization 10MB | English | Russian | Japanese | Chinese |
|---|---|---|---|---|
| fast | 0.1 (x1) | 0.2 (x1) | 0.1 (x1) | 0.1 (x1) |
| hugging face | 4.5 (x52.7) | 5.9 (x38.1) | 4.4 (x52.4) | 5.2 (x63.5) |
| keras | 0.8 (x9.5) | 0.8 (x5.1) | 1.2 (x13.9) | 1.9 (x23.1) |
| linear | 0.8 (x9.8) | 0.5 (x3.4) | 0.3 (x3.9) | 0.4 (x4.3) |
| tensorflow | 5.9 (x69.3) | 3.4 (x22) | 4.0 (x47) | 5.1 (x62.2) |
| torch | 2.2 (x25.5) | 4.4 (x28.4) | 1.9 (x22.7) | 1.3 (x15.6) |
| Tokenization 100MB | English | Russian | Japanese | Chinese |
|---|---|---|---|---|
| fast | 0.7 (x1) | 1.2 (x1) | 0.5 (x1) | 0.4 (x1) |
| hugging face | 47.2 (x65.9) | 61.9 (x51.5) | 49.3 (x109.5) | killed |
| keras | 6.0 (x8.4) | 7.4 (x6.2) | 10.3 (x22.9) | 19.4 (x43.1) |
| linear | 10.6 (x14.9) | 6.6 (x5.5) | 4.3 (x9.5) | 4.4 (x9.8) |
| tensorflow | 56.9 (x79.6) | 33.2 (x27.6) | 38.9 (x86.3) | 50.5 (x112.2) |
| torch | 20.9 (x29.3) | 42.9 (x35.6) | 18.6 (x41.4) | 13.2 (x29.2) |
Mac M1 pro, Apple M1 Pro chip, RAM 16 GB, SSD 512 GB.
| Tokenization 10MB | English | Russian | Japanese | Chinese |
|---|---|---|---|---|
| fast | 0.1 (x1) | 0.2 (x1) | 0.1 (x1) | 0.1 (x1) |
| hugging face | 4.0 (x50.4) | 5.1 (x28.9) | 4.0 (x34.7) | 4.6 (x40.9) |
| linear | 0.4 (x5.1) | 0.3 (x1.6) | 0.2 (x2) | 0.2 (x1.7) |
| torch | 1.1 (x13.7) | 2.2 (x12.3) | 1.2 (x10.9) | 0.9 (x7.7) |
| Tokenization 100MB | English | Russian | Japanese | Chinese |
|---|---|---|---|---|
| fast | 0.6 (x1) | 1.1 (x1) | 0.5 (x1) | 0.5 (x1) |
| hugging face | 60.1 (x94) | 68.0 (x59.6) | 60.8 (x125.6) | 71.1 (x132.7) |
| linear | 5.3 (x8.3) | 3.6 (x3.1) | 2.2 (x4.5) | 2.0 (x3.8) |
| torch | 10.7 (x16.7) | 23.0 (x20.2) | 12.9 (x26.6) | 9.1 (x17) |