Skip to content

Commit

Permalink
nbfmt
Browse files Browse the repository at this point in the history
  • Loading branch information
@bebechien
bebechien committed Jul 31, 2024
1 parent b2557fb commit 68437a2
Showing 1 changed file with 33 additions and 34 deletions.
67 changes: 33 additions & 34 deletions site/en/gemma/docs/lora_tuning.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "tuOe1ymfHZPu"
Expand Down Expand Up @@ -150,7 +150,7 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 1,
"metadata": {
"id": "0_EdOg9DPK6Q"
},
Expand Down Expand Up @@ -179,15 +179,15 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": null,
"metadata": {
"id": "1eeBtYqJsZPG"
},
"outputs": [],
"source": [
"# Install Keras 3 last. See https://keras.io/getting_started/ for more details.\n",
"!pip install -q -U keras-nlp\n",
"!pip install -q -U keras>=3"
"!pip install -q -U \"keras>=3\""
]
},
{
Expand All @@ -205,7 +205,7 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 2,
"metadata": {
"id": "yn5uy8X8sdD0"
},
Expand All @@ -229,7 +229,7 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 3,
"metadata": {
"id": "FYHyPUA9hKTf"
},
Expand All @@ -250,7 +250,7 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 5,
"metadata": {
"id": "xRaNCPUXKoa7"
},
Expand All @@ -259,21 +259,21 @@
"name": "stdout",
"output_type": "stream",
"text": [
"--2024-07-23 07:50:14-- https://huggingface.co/datasets/databricks/databricks-dolly-15k/resolve/main/databricks-dolly-15k.jsonl\n",
"Resolving huggingface.co (huggingface.co)... 3.163.189.74, 3.163.189.114, 3.163.189.37, ...\n",
"Connecting to huggingface.co (huggingface.co)|3.163.189.74|:443... connected.\n",
"--2024-07-31 01:56:39-- https://huggingface.co/datasets/databricks/databricks-dolly-15k/resolve/main/databricks-dolly-15k.jsonl\n",
"Resolving huggingface.co (huggingface.co)... 18.164.174.23, 18.164.174.17, 18.164.174.55, ...\n",
"Connecting to huggingface.co (huggingface.co)|18.164.174.23|:443... connected.\n",
"HTTP request sent, awaiting response... 302 Found\n",
"Location: https://cdn-lfs.huggingface.co/repos/34/ac/34ac588cc580830664f592597bb6d19d61639eca33dc2d6bb0b6d833f7bfd552/2df9083338b4abd6bceb5635764dab5d833b393b55759dffb0959b6fcbf794ec?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27databricks-dolly-15k.jsonl%3B+filename%3D%22databricks-dolly-15k.jsonl%22%3B&Expires=1721980214&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMTk4MDIxNH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jby9yZXBvcy8zNC9hYy8zNGFjNTg4Y2M1ODA4MzA2NjRmNTkyNTk3YmI2ZDE5ZDYxNjM5ZWNhMzNkYzJkNmJiMGI2ZDgzM2Y3YmZkNTUyLzJkZjkwODMzMzhiNGFiZDZiY2ViNTYzNTc2NGRhYjVkODMzYjM5M2I1NTc1OWRmZmIwOTU5YjZmY2JmNzk0ZWM%7EcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj0qIn1dfQ__&Signature=jSlssldDP9jmpAhyuKiR0qLqDgTJTJTHZnkit0dnWJSPoP63bzmGeLfsvMcjJR53aP40nkpBXGugiIYgm5tkyYrqIGXbJeTo3KvK9KqbdMayt0vLKaBvAJwNfPjXEeCuVinTY6thtkH7gARlx9yDVMhY-Btb2jInX1jzxdH2-j2B-B6ipvBCe05ViJFq5l%7EYMjoK1RfrJXqo0gxWR-ktNTxPNZrwRAJQpSJ29MSoefX9x0TxjmYVby8lqe17Y4XNu9IrcDC56ssmaV%7EoGMFNr24pRq7FiewHzh6CXAFnAo1hS26d0H3DR1ZEE1pUkYzGQ-0QjwX67kE8CdVUhUiMHA__&Key-Pair-Id=K3ESJI6DHPFC7 [following]\n",
"--2024-07-23 07:50:14-- https://cdn-lfs.huggingface.co/repos/34/ac/34ac588cc580830664f592597bb6d19d61639eca33dc2d6bb0b6d833f7bfd552/2df9083338b4abd6bceb5635764dab5d833b393b55759dffb0959b6fcbf794ec?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27databricks-dolly-15k.jsonl%3B+filename%3D%22databricks-dolly-15k.jsonl%22%3B&Expires=1721980214&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMTk4MDIxNH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jby9yZXBvcy8zNC9hYy8zNGFjNTg4Y2M1ODA4MzA2NjRmNTkyNTk3YmI2ZDE5ZDYxNjM5ZWNhMzNkYzJkNmJiMGI2ZDgzM2Y3YmZkNTUyLzJkZjkwODMzMzhiNGFiZDZiY2ViNTYzNTc2NGRhYjVkODMzYjM5M2I1NTc1OWRmZmIwOTU5YjZmY2JmNzk0ZWM%7EcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj0qIn1dfQ__&Signature=jSlssldDP9jmpAhyuKiR0qLqDgTJTJTHZnkit0dnWJSPoP63bzmGeLfsvMcjJR53aP40nkpBXGugiIYgm5tkyYrqIGXbJeTo3KvK9KqbdMayt0vLKaBvAJwNfPjXEeCuVinTY6thtkH7gARlx9yDVMhY-Btb2jInX1jzxdH2-j2B-B6ipvBCe05ViJFq5l%7EYMjoK1RfrJXqo0gxWR-ktNTxPNZrwRAJQpSJ29MSoefX9x0TxjmYVby8lqe17Y4XNu9IrcDC56ssmaV%7EoGMFNr24pRq7FiewHzh6CXAFnAo1hS26d0H3DR1ZEE1pUkYzGQ-0QjwX67kE8CdVUhUiMHA__&Key-Pair-Id=K3ESJI6DHPFC7\n",
"Resolving cdn-lfs.huggingface.co (cdn-lfs.huggingface.co)... 108.138.94.14, 108.138.94.122, 108.138.94.23, ...\n",
"Connecting to cdn-lfs.huggingface.co (cdn-lfs.huggingface.co)|108.138.94.14|:443... connected.\n",
"Location: https://cdn-lfs.huggingface.co/repos/34/ac/34ac588cc580830664f592597bb6d19d61639eca33dc2d6bb0b6d833f7bfd552/2df9083338b4abd6bceb5635764dab5d833b393b55759dffb0959b6fcbf794ec?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27databricks-dolly-15k.jsonl%3B+filename%3D%22databricks-dolly-15k.jsonl%22%3B&Expires=1722650199&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMjY1MDE5OX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jby9yZXBvcy8zNC9hYy8zNGFjNTg4Y2M1ODA4MzA2NjRmNTkyNTk3YmI2ZDE5ZDYxNjM5ZWNhMzNkYzJkNmJiMGI2ZDgzM2Y3YmZkNTUyLzJkZjkwODMzMzhiNGFiZDZiY2ViNTYzNTc2NGRhYjVkODMzYjM5M2I1NTc1OWRmZmIwOTU5YjZmY2JmNzk0ZWM%7EcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj0qIn1dfQ__&Signature=nITF8KrgvPBdCRtwfpzGV9ulH2joFLXIDct5Nq-aZqb-Eum8XiVGOai76mxahgAK2mCO4ekuNVCxVsa9Q7h40cZuzViZZC3zAF8QVQlbbkd3FBY4SN3QA4nDNQGcuRYoMKcalA9vRBasFhmdWgupxVqYgMVfJvgSApUcMHMm1HqRBn8AGKpEsaXhEMX4I0N-KtDH5ojDZjz5QBDgkWEmPYUeDQbjVHMjXsRG5z4vH3nK1W9gzC7dkWicJZlzl6iGs44w-EqnD3h-McDCgFnXUacPydm1hdgin-wutx7V4Z3Yv82Fi-TPlDYCnioesUr9Rx8xYujPuXmWP24kPca17Q__&Key-Pair-Id=K3ESJI6DHPFC7 [following]\n",
"--2024-07-31 01:56:39-- https://cdn-lfs.huggingface.co/repos/34/ac/34ac588cc580830664f592597bb6d19d61639eca33dc2d6bb0b6d833f7bfd552/2df9083338b4abd6bceb5635764dab5d833b393b55759dffb0959b6fcbf794ec?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27databricks-dolly-15k.jsonl%3B+filename%3D%22databricks-dolly-15k.jsonl%22%3B&Expires=1722650199&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMjY1MDE5OX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jby9yZXBvcy8zNC9hYy8zNGFjNTg4Y2M1ODA4MzA2NjRmNTkyNTk3YmI2ZDE5ZDYxNjM5ZWNhMzNkYzJkNmJiMGI2ZDgzM2Y3YmZkNTUyLzJkZjkwODMzMzhiNGFiZDZiY2ViNTYzNTc2NGRhYjVkODMzYjM5M2I1NTc1OWRmZmIwOTU5YjZmY2JmNzk0ZWM%7EcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj0qIn1dfQ__&Signature=nITF8KrgvPBdCRtwfpzGV9ulH2joFLXIDct5Nq-aZqb-Eum8XiVGOai76mxahgAK2mCO4ekuNVCxVsa9Q7h40cZuzViZZC3zAF8QVQlbbkd3FBY4SN3QA4nDNQGcuRYoMKcalA9vRBasFhmdWgupxVqYgMVfJvgSApUcMHMm1HqRBn8AGKpEsaXhEMX4I0N-KtDH5ojDZjz5QBDgkWEmPYUeDQbjVHMjXsRG5z4vH3nK1W9gzC7dkWicJZlzl6iGs44w-EqnD3h-McDCgFnXUacPydm1hdgin-wutx7V4Z3Yv82Fi-TPlDYCnioesUr9Rx8xYujPuXmWP24kPca17Q__&Key-Pair-Id=K3ESJI6DHPFC7\n",
"Resolving cdn-lfs.huggingface.co (cdn-lfs.huggingface.co)... 18.154.206.4, 18.154.206.17, 18.154.206.28, ...\n",
"Connecting to cdn-lfs.huggingface.co (cdn-lfs.huggingface.co)|18.154.206.4|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 13085339 (12M) [text/plain]\n",
"Saving to: ‘databricks-dolly-15k.jsonl’\n",
"\n",
"databricks-dolly-15 100%[===================>] 12.48M --.-KB/s in 0.1s \n",
"databricks-dolly-15 100%[===================>] 12.48M 73.7MB/s in 0.2s \n",
"\n",
"2024-07-23 07:50:14 (90.9 MB/s) - ‘databricks-dolly-15k.jsonl’ saved [13085339/13085339]\n",
"2024-07-31 01:56:40 (73.7 MB/s) - ‘databricks-dolly-15k.jsonl’ saved [13085339/13085339]\n",
"\n"
]
}
Expand All @@ -293,7 +293,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 6,
"metadata": {
"id": "ZiS-KU9osh_N"
},
Expand Down Expand Up @@ -330,7 +330,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 4,
"metadata": {
"id": "vz5zLEyLstfn"
},
Expand Down Expand Up @@ -460,8 +460,7 @@
}
],
"source": [
"#gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(\"gemma_2b_en\")\n",
"gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(\"kaggle://google/gemma-2-2b/keras/gemma2_2b_en\")\n",
"gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(\"gemma2_2b_en\")\n",
"gemma_lm.summary()"
]
},
Expand All @@ -471,7 +470,7 @@
"id": "Nl4lvPy5zA26"
},
"source": [
"The `from_preset` method instantiates the model from a preset architecture and weights. In the code above, the string \"gemma_2b_en\" specifies the preset architecture — a Gemma model with 2 billion parameters.\n",
"The `from_preset` method instantiates the model from a preset architecture and weights. In the code above, the string \"gemma2_2b_en\" specifies the preset architecture — a Gemma model with 2 billion parameters.\n",
"\n",
"NOTE: A Gemma model with 7\n",
"billion parameters is also available. To run the larger model in Colab, you need access to the premium GPUs available in paid plans. Alternatively, you can perform [distributed tuning on a Gemma 7B model](https://ai.google.dev/gemma/docs/distributed_tuning) on Kaggle or Google Cloud."
Expand Down Expand Up @@ -501,7 +500,7 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 8,
"metadata": {
"id": "ZwQz3xxxKciD"
},
Expand Down Expand Up @@ -579,7 +578,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 9,
"metadata": {
"id": "lorJMbsusgoo"
},
Expand Down Expand Up @@ -650,7 +649,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 10,
"metadata": {
"id": "RCucu6oHz53G"
},
Expand Down Expand Up @@ -791,12 +790,12 @@
"id": "hQQ47kcdpbZ9"
},
"source": [
"Note that enabling LoRA reduces the number of trainable parameters significantly (from 2.5 billion to 1.3 million)."
"Note that enabling LoRA reduces the number of trainable parameters significantly (from 2.6 billion to 2.9 million)."
]
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 11,
"metadata": {
"id": "_Peq7TnLtHse"
},
Expand All @@ -805,16 +804,16 @@
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1m1000/1000\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m923s\u001b[0m 887ms/step - loss: 1.5585 - sparse_categorical_accuracy: 0.5246\n"
"\u001b[1m1000/1000\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m923s\u001b[0m 888ms/step - loss: 1.5586 - sparse_categorical_accuracy: 0.5251\n"
]
},
{
"data": {
"text/plain": [
"<keras.src.callbacks.history.History at 0x790d47e78f40>"
"<keras.src.callbacks.history.History at 0x799d04393c40>"
]
},
"execution_count": 12,
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
Expand Down Expand Up @@ -854,7 +853,7 @@
},
{
"cell_type": "code",
"execution_count": 13,
"execution_count": null,
"metadata": {
"id": "T0lHxEDX03gp"
},
Expand Down Expand Up @@ -885,7 +884,7 @@
},
{
"cell_type": "code",
"execution_count": 14,
"execution_count": 12,
"metadata": {
"id": "Y7cDJHy8WfCB"
},
Expand All @@ -898,7 +897,7 @@
"What should I do on a trip to Europe?\n",
"\n",
"Response:\n",
"I would take my laptop and a portable charger, and I would download all of the maps and guides I need for my trip. I would also download the local news and weather app, which I can use to stay updated on the weather and any important news events in the area. I would also download the language learning app so that I can learn a few phrases in the local language.\n"
"When planning a trip to Europe, you should consider your budget, time and the places you want to visit. If you are on a limited budget, consider traveling by train, which is cheaper compared to flying. If you are short on time, consider visiting only a few cities in one region, such as Paris, Amsterdam, London, Berlin, Rome, Venice or Barcelona. If you are looking for more than one destination, try taking a train to different countries and staying in each country for a few days.\n"
]
}
],
Expand Down Expand Up @@ -932,7 +931,7 @@
},
{
"cell_type": "code",
"execution_count": 15,
"execution_count": 13,
"metadata": {
"id": "X-2sYl2jqwl7"
},
Expand All @@ -945,7 +944,7 @@
"Explain the process of photosynthesis in a way that a child could understand.\n",
"\n",
"Response:\n",
"Plants use sunlight, carbon dioxide, and water to produce glucose and release oxygen into the atmosphere. The process of photosynthesis is carried out by the plant's green leaves, which contain the green pigment chlorophyll. During the process of photosynthesis, the plant absorbs carbon dioxide from the air and water from the roots. It then uses sunlight energy to convert water into oxygen. The oxygen is released into the air, while the plant uses the glucose to grow.\n"
"The process of photosynthesis is a chemical reaction in plants that converts the energy of sunlight into chemical energy, which the plants can then use to grow and develop. During photosynthesis, a plant will absorb carbon dioxide (CO2) from the air and water from the soil and use the energy from the sun to produce oxygen (O2) and sugars (glucose) as a by-product.\n"
]
}
],
Expand Down

0 comments on commit 68437a2

Please sign in to comment.