From c8b96b4dc234786280840e070f5e9facb24d2770 Mon Sep 17 00:00:00 2001 From: andrius-meta Date: Fri, 19 Jul 2024 20:09:57 +0200 Subject: [PATCH] Adding an example for TorchServe to use `torch.compile` for a (#3259) batch `Resnet152` model, similar to what already exists for `densenet161`. - The documentation of batched `Resnet152` was updated. - [x] Did you have fun? - [ ] Have you added tests that prove your fix is effective or that this feature works? - [ ] Has code been commented, particularly in hard-to-understand areas? - [x] Have you made corresponding changes to the documentation? Co-authored-by: Ankith Gunapal --- .../resnet_152_batch/README.md | 31 ++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/examples/image_classifier/resnet_152_batch/README.md b/examples/image_classifier/resnet_152_batch/README.md index 7f5e7341dd..327fa7c31c 100644 --- a/examples/image_classifier/resnet_152_batch/README.md +++ b/examples/image_classifier/resnet_152_batch/README.md @@ -1,3 +1,32 @@ +#### TorchServe inference with torch.compile of Resnet152 batch image classifier: +Run the commands given in following steps from the root directory of the repository. For example, if you cloned the repository into /home/my_path/serve, run the steps from /home/my_path/serve + +First, we create a configuration file where we enable `torch.compile`. You can find more information about the available configuration options [here](https://pytorch.org/docs/stable/generated/torch.compile.html). + +```bash +echo "pt2: + compile: + enable: True" > examples/image_classifier/resnet_152_batch/model-config.yaml +``` + +Sample commands to create a Resnet152 torch.compile model archive for batch inputs, register it on TorchServe and run image prediction: + +```bash +wget https://download.pytorch.org/models/resnet152-394f9c45.pth +torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --config-file examples/image_classifier/resnet_152_batch/model-config.yaml +mkdir model-store +mv resnet-152-batch.mar model-store/ +torchserve --start --model-store model-store --disable-token-auth --enable-model-api +curl -X POST "localhost:8081/models?model_name=resnet152&url=resnet-152-batch.mar&batch_size=4&max_batch_delay=5000&initial_workers=3&synchronous=true" +``` + +And run batched inference: +```bash +curl http://localhost:8080/predictions/resnet152 -T examples/image_classifier/resnet_152_batch/images/croco.jpg & +curl http://localhost:8080/predictions/resnet152 -T examples/image_classifier/resnet_152_batch/images/dog.jpg & +curl http://localhost:8080/predictions/resnet152 -T examples/image_classifier/resnet_152_batch/images/kitten.jpg & +``` + #### Sample commands to create a resnet-152 eager mode model archive for batch inputs, register it on TorchServe and run image prediction Run the commands given in following steps from the root directory of the repository. For example, if you cloned the repository into /home/my_path/serve, run the steps from /home/my_path/serve @@ -6,7 +35,7 @@ wget https://download.pytorch.org/models/resnet152-394f9c45.pth torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json mkdir model-store mv resnet-152-batch.mar model-store/ -torchserve --start --model-store model-store --disable-token-auth --enable-model-api +torchserve --start --model-store model-store --disable-token-auth --enable-model-api curl -X POST "localhost:8081/models?model_name=resnet152&url=resnet-152-batch.mar&batch_size=4&max_batch_delay=5000&initial_workers=3&synchronous=true" ```