Skip to content

Commit

Permalink
Adding an example for TorchServe to use torch.compile for a (#3259)
Browse files Browse the repository at this point in the history
batch `Resnet152` model, similar to what already exists for `densenet161`.

- The documentation of batched `Resnet152` was updated.

- [x] Did you have fun?
- [ ] Have you added tests that prove your fix is effective or that this feature works?
- [ ] Has code been commented, particularly in hard-to-understand areas?
- [x] Have you made corresponding changes to the documentation?

Co-authored-by: Ankith Gunapal <agunapal@ischool.Berkeley.edu>
  • Loading branch information
andrius-meta and agunapal authored Jul 19, 2024
1 parent b985beb commit c8b96b4
Showing 1 changed file with 30 additions and 1 deletion.
31 changes: 30 additions & 1 deletion examples/image_classifier/resnet_152_batch/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,32 @@
#### TorchServe inference with torch.compile of Resnet152 batch image classifier:
Run the commands given in following steps from the root directory of the repository. For example, if you cloned the repository into /home/my_path/serve, run the steps from /home/my_path/serve

First, we create a configuration file where we enable `torch.compile`. You can find more information about the available configuration options [here](https://pytorch.org/docs/stable/generated/torch.compile.html).

```bash
echo "pt2:
compile:
enable: True" > examples/image_classifier/resnet_152_batch/model-config.yaml
```

Sample commands to create a Resnet152 torch.compile model archive for batch inputs, register it on TorchServe and run image prediction:

```bash
wget https://download.pytorch.org/models/resnet152-394f9c45.pth
torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --config-file examples/image_classifier/resnet_152_batch/model-config.yaml
mkdir model-store
mv resnet-152-batch.mar model-store/
torchserve --start --model-store model-store --disable-token-auth --enable-model-api
curl -X POST "localhost:8081/models?model_name=resnet152&url=resnet-152-batch.mar&batch_size=4&max_batch_delay=5000&initial_workers=3&synchronous=true"
```

And run batched inference:
```bash
curl http://localhost:8080/predictions/resnet152 -T examples/image_classifier/resnet_152_batch/images/croco.jpg &
curl http://localhost:8080/predictions/resnet152 -T examples/image_classifier/resnet_152_batch/images/dog.jpg &
curl http://localhost:8080/predictions/resnet152 -T examples/image_classifier/resnet_152_batch/images/kitten.jpg &
```

#### Sample commands to create a resnet-152 eager mode model archive for batch inputs, register it on TorchServe and run image prediction
Run the commands given in following steps from the root directory of the repository. For example, if you cloned the repository into /home/my_path/serve, run the steps from /home/my_path/serve

Expand All @@ -6,7 +35,7 @@ wget https://download.pytorch.org/models/resnet152-394f9c45.pth
torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json
mkdir model-store
mv resnet-152-batch.mar model-store/
torchserve --start --model-store model-store --disable-token-auth --enable-model-api
torchserve --start --model-store model-store --disable-token-auth --enable-model-api
curl -X POST "localhost:8081/models?model_name=resnet152&url=resnet-152-batch.mar&batch_size=4&max_batch_delay=5000&initial_workers=3&synchronous=true"
```

Expand Down

0 comments on commit c8b96b4

Please sign in to comment.