diff --git a/examples/image_classifier/resnet_152_batch/README.md b/examples/image_classifier/resnet_152_batch/README.md index 7f5e7341dd..327fa7c31c 100644 --- a/examples/image_classifier/resnet_152_batch/README.md +++ b/examples/image_classifier/resnet_152_batch/README.md @@ -1,3 +1,32 @@ +#### TorchServe inference with torch.compile of Resnet152 batch image classifier: +Run the commands given in following steps from the root directory of the repository. For example, if you cloned the repository into /home/my_path/serve, run the steps from /home/my_path/serve + +First, we create a configuration file where we enable `torch.compile`. You can find more information about the available configuration options [here](https://pytorch.org/docs/stable/generated/torch.compile.html). + +```bash +echo "pt2: + compile: + enable: True" > examples/image_classifier/resnet_152_batch/model-config.yaml +``` + +Sample commands to create a Resnet152 torch.compile model archive for batch inputs, register it on TorchServe and run image prediction: + +```bash +wget https://download.pytorch.org/models/resnet152-394f9c45.pth +torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --config-file examples/image_classifier/resnet_152_batch/model-config.yaml +mkdir model-store +mv resnet-152-batch.mar model-store/ +torchserve --start --model-store model-store --disable-token-auth --enable-model-api +curl -X POST "localhost:8081/models?model_name=resnet152&url=resnet-152-batch.mar&batch_size=4&max_batch_delay=5000&initial_workers=3&synchronous=true" +``` + +And run batched inference: +```bash +curl http://localhost:8080/predictions/resnet152 -T examples/image_classifier/resnet_152_batch/images/croco.jpg & +curl http://localhost:8080/predictions/resnet152 -T examples/image_classifier/resnet_152_batch/images/dog.jpg & +curl http://localhost:8080/predictions/resnet152 -T examples/image_classifier/resnet_152_batch/images/kitten.jpg & +``` + #### Sample commands to create a resnet-152 eager mode model archive for batch inputs, register it on TorchServe and run image prediction Run the commands given in following steps from the root directory of the repository. For example, if you cloned the repository into /home/my_path/serve, run the steps from /home/my_path/serve @@ -6,7 +35,7 @@ wget https://download.pytorch.org/models/resnet152-394f9c45.pth torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json mkdir model-store mv resnet-152-batch.mar model-store/ -torchserve --start --model-store model-store --disable-token-auth --enable-model-api +torchserve --start --model-store model-store --disable-token-auth --enable-model-api curl -X POST "localhost:8081/models?model_name=resnet152&url=resnet-152-batch.mar&batch_size=4&max_batch_delay=5000&initial_workers=3&synchronous=true" ```