Skip to content

Commit

Permalink
feat(docs): [SCv2] Add a section about loading custom HuggingFace mod…
Browse files Browse the repository at this point in the history 
…els from Seldon CLI (#5106)

* Add docs for usage of Seldon CLI to load custom HF models

* Update GCS location to custom HF model
  • Loading branch information
@vtaskow
vtaskow authored Aug 29, 2023
1 parent 2b7d365 commit 564a2cd
Show file tree
Hide file tree
Showing 3 changed files with 267 additions and 17 deletions.
183 changes: 170 additions & 13 deletions samples/huggingface.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@
"{\r\n",
"\t\"model_name\": \"text-gen_1\",\r\n",
"\t\"model_version\": \"1\",\r\n",
"\t\"id\": \"bde3e68b-2710-4763-828c-21e58416b45c\",\r\n",
"\t\"id\": \"bcd24642-ade5-4b39-9c70-5d62bbfbe43c\",\r\n",
"\t\"parameters\": {},\r\n",
"\t\"outputs\": [\r\n",
"\t\t{\r\n",
Expand All @@ -109,10 +109,10 @@
"\t\t\t],\r\n",
"\t\t\t\"datatype\": \"BYTES\",\r\n",
"\t\t\t\"parameters\": {\r\n",
"\t\t\t\t\"content_type\": \"str\"\r\n",
"\t\t\t\t\"content_type\": \"hg_jsonlist\"\r\n",
"\t\t\t},\r\n",
"\t\t\t\"data\": [\r\n",
"\t\t\t\t\"[{\\\"generated_text\\\": \\\"Once upon a time in a galaxy far away, you can see the galaxy and the universe together.\\\\n\\\\n\\\\n\\\\nIn the same space as Earth, there are two locations (including Earth) or regions. These regions are located on both of the\\\"}]\"\r\n",
"\t\t\t\t\"{\\\"generated_text\\\": \\\"Once upon a time in a galaxy far away, the Galaxy has made it a little easier to travel to and from your home planet through the galaxy's solar system. The planet's atmosphere is also a key asset to the Galaxy's galactic evolution! The\\\"}\"\r\n",
"\t\t\t]\r\n",
"\t\t}\r\n",
"\t]\r\n",
Expand All @@ -127,7 +127,7 @@
},
{
"cell_type": "code",
"execution_count": 17,
"execution_count": 5,
"id": "right-talent",
"metadata": {},
"outputs": [],
Expand All @@ -138,17 +138,17 @@
},
{
"cell_type": "code",
"execution_count": 18,
"execution_count": 6,
"id": "4a7d7108",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"b'[{\"generated_text\": \"Once upon a time\\\\n\\\\n\\\\nThe Great Depression was a devastating economic collapse. The depression was not only due to soaring interest rates, but to an epidemic of illegal foreign trade. It was the worst recession of all time because of the government\\'s actions\"}]'"
"b'{\"generated_text\": \"Once upon a time\\\\n\\\\n\\\\nI have no idea if he or she will fall prey to my whims\\\\nAnd\\\\n\\\\nAnd when\\\\n\\\\nOr\\\\n\\\\nAnd when I say\\\\nTo\\\\n\\\\nI\\'ll have to\\\\nEven\\\\nOr\\\\nTo\"}'"
]
},
"execution_count": 18,
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -170,9 +170,61 @@
},
{
"cell_type": "code",
"execution_count": 19,
"execution_count": 7,
"id": "ceramic-illness",
"metadata": {},
"outputs": [],
"source": [
"!seldon model unload text-gen"
]
},
{
"cell_type": "markdown",
"id": "6ff63bb7",
"metadata": {},
"source": [
"### Custom Text Generation Model\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "6d48e38c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"apiVersion: mlops.seldon.io/v1alpha1\r\n",
"kind: Model\r\n",
"metadata:\r\n",
" name: custom-text-gen\r\n",
"spec:\r\n",
" storageUri: \"gs://seldon-models/scv2/samples/mlserver_1.3.5/custom-text-generation-huggingface\"\r\n",
" requirements:\r\n",
" - huggingface\r\n",
" memory: 3Gi\r\n"
]
}
],
"source": [
"!cat ./models/hf-custom-text-gen.yaml"
]
},
{
"cell_type": "markdown",
"id": "62ec637f",
"metadata": {},
"source": [
"Load the model"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "abef8088",
"metadata": {},
"outputs": [
{
"name": "stdout",
Expand All @@ -183,16 +235,121 @@
}
],
"source": [
"!seldon model unload text-gen"
"!seldon model load -f ./models/hf-custom-text-gen.yaml"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3a29f9cb",
"execution_count": 10,
"id": "f25c54d4",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{}\r\n"
]
}
],
"source": [
"!seldon model status custom-text-gen -w ModelAvailable | jq -M ."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "21f26bbd",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\r\n",
"\t\"model_name\": \"custom-text-gen_1\",\r\n",
"\t\"model_version\": \"1\",\r\n",
"\t\"id\": \"ea5d4209-a5cf-4179-ba81-87b8bcfdf894\",\r\n",
"\t\"parameters\": {},\r\n",
"\t\"outputs\": [\r\n",
"\t\t{\r\n",
"\t\t\t\"name\": \"output\",\r\n",
"\t\t\t\"shape\": [\r\n",
"\t\t\t\t1,\r\n",
"\t\t\t\t1\r\n",
"\t\t\t],\r\n",
"\t\t\t\"datatype\": \"BYTES\",\r\n",
"\t\t\t\"parameters\": {\r\n",
"\t\t\t\t\"content_type\": \"hg_jsonlist\"\r\n",
"\t\t\t},\r\n",
"\t\t\t\"data\": [\r\n",
"\t\t\t\t\"{\\\"generated_text\\\": \\\"Once upon a time in a galaxy far away, scientists in the universe are told the universe is on a collision course with stars, and the result is the emergence of new universes. This has been possible for, say, the distant history of the Milky\\\"}\"\r\n",
"\t\t\t]\r\n",
"\t\t}\r\n",
"\t]\r\n",
"}\r\n"
]
}
],
"source": [
"!seldon model infer custom-text-gen \\\n",
" '{\"inputs\": [{\"name\": \"args\",\"shape\": [1],\"datatype\": \"BYTES\",\"data\": [\"Once upon a time in a galaxy far away\"]}]}'"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "30e8bb29",
"metadata": {},
"outputs": [],
"source": [
"res = !seldon model infer custom-text-gen --inference-mode grpc \\\n",
" '{\"inputs\":[{\"name\":\"args\",\"contents\":{\"bytes_contents\":[\"T25jZSB1cG9uIGEgdGltZQo=\"]},\"datatype\":\"BYTES\",\"shape\":[1]}]}' "
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "eab4929b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"b'{\"generated_text\": \"Once upon a time\\\\n\\\\nand again\\\\n\\\\nwe took care not a minute of\\\\n\\\\nbeing alone in a\\\\n\\\\nworld that we had not seen\\\\n\\\\nfor\\\\n\\\\nten thousand years\\\\n\\\\nbefore it occurred to our\\\\n\\\\nmind that\"}'"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import json\n",
"import base64\n",
"r = json.loads(res[0])\n",
"base64.b64decode(r[\"outputs\"][0][\"contents\"][\"bytesContents\"][0])"
]
},
{
"cell_type": "markdown",
"id": "877e266c",
"metadata": {},
"source": [
"Unload the model"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "e1829439",
"metadata": {},
"outputs": [],
"source": []
"source": [
"!seldon model unload custom-text-gen"
]
}
],
"metadata": {
Expand All @@ -211,7 +368,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.13"
"version": "3.10.12"
}
},
"nbformat": 4,
Expand Down
92 changes: 88 additions & 4 deletions samples/huggingface.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ seldon model infer text-gen \
{
"model_name": "text-gen_1",
"model_version": "1",
"id": "bde3e68b-2710-4763-828c-21e58416b45c",
"id": "bcd24642-ade5-4b39-9c70-5d62bbfbe43c",
"parameters": {},
"outputs": [
{
Expand All @@ -58,10 +58,10 @@ seldon model infer text-gen \
],
"datatype": "BYTES",
"parameters": {
"content_type": "str"
"content_type": "hg_jsonlist"
},
"data": [
"[{\"generated_text\": \"Once upon a time in a galaxy far away, you can see the galaxy and the universe together.\\n\\n\\n\\nIn the same space as Earth, there are two locations (including Earth) or regions. These regions are located on both of the\"}]"
"{\"generated_text\": \"Once upon a time in a galaxy far away, the Galaxy has made it a little easier to travel to and from your home planet through the galaxy's solar system. The planet's atmosphere is also a key asset to the Galaxy's galactic evolution! The\"}"
]
}
]
Expand All @@ -82,7 +82,7 @@ base64.b64decode(r["outputs"][0]["contents"]["bytesContents"][0])
```

```
b'[{"generated_text": "Once upon a time\\n\\n\\nThe Great Depression was a devastating economic collapse. The depression was not only due to soaring interest rates, but to an epidemic of illegal foreign trade. It was the worst recession of all time because of the government\'s actions"}]'
b'{"generated_text": "Once upon a time\\n\\n\\nI have no idea if he or she will fall prey to my whims\\nAnd\\n\\nAnd when\\n\\nOr\\n\\nAnd when I say\\nTo\\n\\nI\'ll have to\\nEven\\nOr\\nTo"}'
```

Expand All @@ -92,11 +92,95 @@ Unload the model
seldon model unload text-gen
```

### Custom Text Generation Model

```bash
cat ./models/hf-custom-text-gen.yaml
```

```yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
name: custom-text-gen
spec:
storageUri: "gs://seldon-models/scv2/samples/mlserver_1.3.5/custom-text-generation-huggingface"
requirements:
- huggingface
memory: 3Gi

```

Load the model

```bash
seldon model load -f ./models/hf-custom-text-gen.yaml
```

```json
{}

```

```bash
seldon model status custom-text-gen -w ModelAvailable | jq -M .
```

```json
{}

```

```bash
seldon model infer custom-text-gen \
'{"inputs": [{"name": "args","shape": [1],"datatype": "BYTES","data": ["Once upon a time in a galaxy far away"]}]}'
```

```json
{
"model_name": "custom-text-gen_1",
"model_version": "1",
"id": "ea5d4209-a5cf-4179-ba81-87b8bcfdf894",
"parameters": {},
"outputs": [
{
"name": "output",
"shape": [
1,
1
],
"datatype": "BYTES",
"parameters": {
"content_type": "hg_jsonlist"
},
"data": [
"{\"generated_text\": \"Once upon a time in a galaxy far away, scientists in the universe are told the universe is on a collision course with stars, and the result is the emergence of new universes. This has been possible for, say, the distant history of the Milky\"}"
]
}
]
}

```

```python
res = !seldon model infer custom-text-gen --inference-mode grpc \
'{"inputs":[{"name":"args","contents":{"bytes_contents":["T25jZSB1cG9uIGEgdGltZQo="]},"datatype":"BYTES","shape":[1]}]}'
```

```python
import json
import base64
r = json.loads(res[0])
base64.b64decode(r["outputs"][0]["contents"]["bytesContents"][0])
```

```
b'{"generated_text": "Once upon a time\\n\\nand again\\n\\nwe took care not a minute of\\n\\nbeing alone in a\\n\\nworld that we had not seen\\n\\nfor\\n\\nten thousand years\\n\\nbefore it occurred to our\\n\\nmind that"}'
```

Unload the model

```bash
seldon model unload custom-text-gen
```
9 changes: 9 additions & 0 deletions samples/models/hf-custom-text-gen.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
name: custom-text-gen
spec:
storageUri: "gs://seldon-models/scv2/samples/mlserver_1.3.5/custom-text-generation-huggingface"
requirements:
- huggingface
memory: 3Gi

0 comments on commit 564a2cd

Please sign in to comment.