From ed83a03e7930564f11c24537294b3988b9584ed5 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang <33082367+IsakZhang@users.noreply.github.com> Date: Tue, 9 Jul 2024 22:36:24 +0800 Subject: [PATCH] Update index.html for SeaLLM3 --- index.html | 894 ++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 608 insertions(+), 286 deletions(-) diff --git a/index.html b/index.html index 2de86d1..ea06642 100644 --- a/index.html +++ b/index.html @@ -6,7 +6,7 @@ content="SeaLLMs - Large Language Models for Southeast Asia"> - SeaLLMs (v2.5) - Large Language Models for Southeast Asia + SeaLLMs - Large Language Models for Southeast Asia @@ -41,10 +41,7 @@ - + @@ -151,7 +148,7 @@

- 🤗 @@ -171,7 +168,7 @@

- @@ -205,19 +202,25 @@

🔥[NEW!] - SeaLLM-7B-v2.5 is released with SoTA in world knowledge and math reasoning. + SeaLLM3 is released with SoTA performance in various tasks and specifically enhanced to be more trustworthy.
+

- + + +
-
@@ -226,9 +229,7 @@

Abstract

- We introduce SeaLLM-7B-v2.5, the state-of-the-art multilingual LLM for Southeast Asian (SEA) languages 🇬🇧 🇨🇳 🇻🇳 🇮🇩 🇹🇭 🇲🇾 🇰🇭 🇱🇦 🇲🇲 🇵🇭. - It outperforms comparable baselines across diverse multilingual tasks, from world knowledge, math reasoning, instruction following, etc. - It also surpasses ChatGPT-3.5 in various knowledge and reasoning bechmarks in multiple non-Latin languages (Thai, Khmer, Lao and Burmese), while remaining lightweight and open-source. + We introduce SeaLLM3, the latest series of the SeaLLMs (Large Language Models for Southeast Asian languages) family. It achieves state-of-the-art performance among models with similar sizes, excelling across a diverse array of tasks such as world knowledge, mathematical reasoning, translation, and instruction following. In the meantime, it was specifically enhanced to be more trustworthy, exhibiting reduced hallucination and providing safe responses, particularly in queries closed related to Southeast Asian culture.

SeaLLMs is a continuously iterated and improved series of language models @@ -264,100 +265,122 @@

SeaLLM-7B-v2.5 DEMO

-

World Knowledge

+

Multilingual World Knowledge - M3Exam

-

- We evaluate models on 3 benchmarks following the recommended default setups: 5-shot MMLU for Eng, 3-shot M3Exam - for Eng, Zho, Vie, Ind, Tha, and zero-shot VMLU for Vie. -

-

- M3Exam was evaluated using the standard prompting implementation, - while 0-shot VMLU was run with vmlu_run.py for SeaLLMs. -

+

M3Exam consists of local exam questions collected from each country. It reflects the model's world knowledge (e.g., with language or social science subjects) and reasoning abilities (e.g., with mathematics or natural science subjects).

+
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelLangsEng
MMLU
5 shots		Eng
M3exam
3 shots		Zho
M3exam
3 shots		Vie
M3exam
3 shots		Vie
VMLU
0 shots		Ind
M3exam
3 shots		Tha
M3exam
3 shots
ChatGPT-3.5Multi68.9075.4660.2058.6446.3249.2737.41
Vistral-7B-chatMono56.8667.0044.5654.3350.0336.4925.27
Qwen1.5-7B-chatMulti61.0052.0781.9643.3845.0224.2920.25
SailorLM-7BMulti52.7259.7667.7450.14 --- 39.5337.73
SeaLLM-7B-v2Multi61.8970.9155.4351.1545.7442.2535.52
SeaLLM-7B-v2.5Multi64.0576.8762.5463.1153.3048.6446.86
Modelenzhidthviavgavg_sea
Sailor-7B-Chat0.660.6520.4750.4620.5130.5520.483
gemma-7b0.7320.5190.4750.460.5940.5560.510
SeaLLM-7B-v2.50.7580.5810.4990.5020.6220.5920.541
Qwen2-7B0.8150.8740.530.4790.6280.6650.546
Qwen2-7B-Instruct0.8090.880.5580.5550.6240.6850.579
Sailor-14B0.7480.840.5360.5280.6210.6550.562
Sailor-14B-Chat0.7490.8430.5530.5660.6370.670.585
SeaLLM3-7B0.8140.8660.5490.520.6280.6750.566
SeaLLM3-7B-Chat0.8090.8740.5580.5690.6490.6920.592
+ + -

Multilingual Math Reasoning

+ --> + + +

Multilingual Instruction-following Capability - SeaBench

-

- SeaLLM-7B-v2.5 achieves with 78.5 and 34.9 in GSM8K and MATH with zero-shot CoT reasoning, making it outperforms GPT-3.5 in MATH. - It also outperforms GPT-3.5 in all GSM8K and MATH benchmark as translated into 4 SEA languages (🇨🇳 🇻🇳 🇮🇩 🇹🇭). -

+

SeaBench consists of multi-turn human instructions spanning various task types. It evaluates chat-based models on their ability to follow human instructions in both single and multi-turn settings and assesses their performance across different task types. The dataset and corresponding evaluation code will be released soon!

+
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelEng
GSM8K		Eng
MATH		Zho
GSM8K		Zho
MATH		Vie
GSM8K		Vie
MATH		Ind
GSM8K		Ind
MATH		Tha
GSM8K		Tha
MATH
ChatGPT-3.580.834.148.221.555.026.564.326.435.818.1
Vistral-7B-Chat48.212.548.73.1
Qwen1.5-7B-chat56.815.340.02.737.79.036.97.721.94.7
SeaLLM-7B-v278.227.553.717.669.923.871.524.459.622.4
SeaLLM-7B-v2.578.534.951.322.172.330.271.530.162.028.4
Modelid
turn1		id
turn2		id
avg		th
turn1		th
turn2		th
avg		vi
turn1		vi
turn2		vi
avg		avg
Qwen2-7B-Instruct5.935.845.895.475.205.346.175.605.895.70
SeaLLM-7B-v2.56.274.965.625.793.824.816.024.025.025.15
Sailor-14B-Chat5.265.535.404.624.364.495.314.745.034.97
Sailor-7B-Chat4.604.044.323.943.173.564.823.624.224.03
SeaLLM3-7B-Chat6.736.596.666.485.906.196.345.796.076.31
- -

Multilingual Instruction Following

+ +

Multilingual Math

-

- Sea-Bench is a set of categorized instruction test sets to measure models' ability as an assistant that is specifically focused on 9 SEA languages, - including non-Latin low-resource languages. Sea-Bench's model responses are rated by GPT-4 following MT-bench LLM-judge procedure. -
- As shown, SeaLLM-7B-v2.5 reaches GPT-3.5 level of performance in many common SEA languages (Eng, Zho, Vie, Ind, Tha, Msa) - and far-surpasses it in low-resource non-Latin languages (Mya, Lao, Khm). -

- - +

We evaluate the multilingual math capability using the MGSM dataset. MGSM originally contains Chinese and Thai testing sets only, we use Google Translate to translate the same English questions into other SEA languages. Note that we adopt the tradition of each country to represent the number, e.g., in Indonesian and Vietnamese, dots are used as thousands separators and commas as decimal separators, the opposite of the English system.

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MGSMenidmsthvizhavg
Sailor-7B-Chat33.622.422.421.625.229.225.7
Meta-Llama-3-8B-Instruct77.64857.65646.858.857.5
glm-4-9b-chat72.853.653.634.852.470.856.3
Qwen1.5-7B-Chat6434.438.425.23653.641.9
Qwen2-7B-instruct8266.462.458.464.476.868.4
aya-23-8B28.816.414.421612.815.1
gemma-1.1-7b-it58.832.434.831.239.635.238.7
SeaLLM-7B-v2.579.669.270.861.266.862.468.3
SeaLLM3-7B-Chat74.871.270.871.271.279.673.1
+
- -

Zero-shot Commonsense Reasoning

+ +

Translation

-

- We compare SeaLLM-7B-v2.5 with ChatGPT and Mistral-7B-instruct on various zero-shot commonsense benchmarks (Arc-Challenge, Winogrande and Hellaswag). We use the 2-stage technique in (Kojima et al., 2023) to grab the answer. Note that we DID NOT use "Let's think step-by-step" to invoke explicit CoT. -

+

We use the test sets from Flores-200 for evaluation and report the zero-shot chrF scores for translations between every pair of languages. Each row in the table below presents the average results of translating from various source languages into the target languages. The last column displays the overall average results of translating from any language to any other language for each model.

+
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelArc-ChallengeWinograndeHellaswag
ChatGPT (Reported)84.6*66.8*72.0*
ChatGPT (Reproduced)84.163.179.5
Mistral-7B-Instruct68.156.445.6
Qwen1.5-7B-Chat79.359.469.3
SeaLLM-7B-v282.568.380.9
SeaLLM-7B-v2.586.575.491.6
Modelenidjvkmlomsmytathtlvizhavg
Meta-Llama-3-8B-Instruct51.5449.0322.4615.345.4246.7221.2432.0935.7540.8039.3114.8731.22
Qwen2-7B-Instruct50.3647.5529.3619.2611.0642.4319.3320.0436.0737.9139.6322.8731.32
Sailor-7B-Chat49.4049.7828.332.686.8547.755.3518.2338.9229.0041.7620.8728.24
SeaLLM-7B-v2.555.0953.7118.1318.0915.5351.3319.7126.1040.5545.5844.5624.1834.38
SeaLLM3-7B-Chat54.6852.5229.8627.3026.3445.0421.5431.9341.5238.5143.7826.1036.52
-
-
+

Hallucination

+
+

Performance of whether a model can refuse questions about the non-existing entity. The following is the F1 score. We use refusal as the positive label. Our test set consists of ~1k test samples per language. Each unanswerable question is generated by GPT4o. The ratio of answerable and unanswerable questions are 1:1. We define keywords to automatically detect whether a model-generated response is a refusal response.

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Refusal-F1 Scoresenzhvithidavg
Qwen1.5-7B-Instruct53.8551.7052.8535.5058.4050.46
Qwen2-7B-Instruct58.7933.0856.2144.6055.9849.732
SeaLLM-7B-v2.512.900.772.4519.420.787.26
Sailor-7B-Chat33.4918.825.199.6816.4216.72
glm-4-9b-chat44.4837.8918.664.271.9721.45
aya-23-8B6.380.792.831.9814.805.36
Llama-3-8B-Instruct72.080.001.230.803.9115.60
gemma-1.1-7b-it52.3927.7423.9622.9731.7231.76
SeaLLM3-7B-Chat71.3678.3977.9361.3168.9571.588
+
+
- -
-
- -

Model Information

+

Safety

-

- All SeaLLM models underwent continue-pretraining, instruction and alignment tuning to - ensure not only their competitive performances in SEA languages, but also maintain high level of safety and legal compliance. - All models are trained with 32 A800 GPUs. -

+

Multijaildataset consists of harmful prompts in multiple languages. We take those relevant prompts in SEA languages here and report their safe rate (the higher the better).

+
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelBackboneContext LengthVocab SizeChat format
SeaLLM-7B-v2.5gemma-7b8192256000 - Add <bos> at start if your tokenizer does not do so! - 
<|im_start|>user
-{content}<eos>
-<|im_start|>assistant
-{content}<eos>
SeaLLM-7B-v2Mistral-7B-v0.1819248384 - Add <bos> at start if your tokenizer does not do so! - 
<|im_start|>user
-{content}</s><|im_start|>assistant
-{content}</s>
SeaLLM-7B-v1Llama-2-7b409648512Same as Llama-2
Modelenjvthvizhavg
Qwen2-7B-Instruct0.88570.43810.63810.73020.8730.713
Sailor-7B-Chat0.78730.54920.62220.67620.76190.6794
Meta-Llama-3-8B-Instruct0.88250.26350.71110.69840.77140.6654
Sailor-14B-Chat0.86980.30480.53650.60950.7270.6095
glm-4-9b-chat0.77140.21270.30160.60630.74920.52824
SeaLLM3-7B-Chat0.88890.60000.73330.83810.9270.7975
@@ -577,10 +891,18 @@

Model Information

+ + + +

Related Links

+

+ SeaLLM3 was released in July 2024. It achieves SOTA performance of diverse tasks while specifically enhanced to be more trustworthy, exhibiting reduced hallucination and providing safe response. +

+

SeaLLM-7B-v2.5 was released in April 2024. It possesses outstanding abilities in world knowledge and math reasoning in both English and SEA languages.