Triton inference server and inference on different machines.

[prasad-nair]

Is it possible to run Triton on one device and do inferences of models on other devices.

[ApoorveK]

@prasad-nair You might need a container orchestration tool setup (Kubernetes) where you can have instances of Triton on different devices which is being managed as worker node by kubernetes master node server. Apart from it, other ways would add latency to inference time hence might not be useful. (but like to know more about them)

[okyspace]

I recalled we might be able to add more than one model repos to the same triton server.

With this if u are hosting models in another device and able to mount as one repo in triton, it might be possible?