You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently we have installed Seldon Core v2 with version 2.6.0 on a k8s cluster. Based on the installation instructions of kafka on our k8s cluster, we followed the recommendation using Strimzi Operator with all default values from helm chart strimzi-kafka-operator (version 0.35.1) and all default values from seldon's helm chart seldon-core-v2-kafka (version 0.1.0). All control plane and data plane operations are running with the security protocol PLAINTEXT.
The goal of this issue is to find out what is the best solution to inspect output data from each model within a Pipeline.
Describe the bug
We built the seldon cli image on our machine and we created a custom configuration on /home/.config/seldon/cli with the following:
We can make an inference through seldon cli, but when we try to do the command seldon pipeline inspect tfsimples the following error appears on console:
%3|1703095841.652|FAIL|rdkafka#consumer-1| [thrd:seldon-kafka-0.seldon-kafka-brokers.seldon-mesh.svc:9092/0]: seldon-kafka-0.seldon-kafka-brokers.seldon-mesh.svc:9092/0: Failed to resolve 'seldon-kafka-0.seldon-kafka-brokers.seldon-mesh.svc:9092': nodename nor servname provided, or not known (after 2ms in state CONNECT)
%3|1703095841.652|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: seldon-kafka-0.seldon-kafka-brokers.seldon-mesh.svc:9092: Failed to resolve 'seldon-kafka-0.seldon-kafka-brokers.seldon-mesh.svc:9092': nodename nor servname provided, or not known (after 2ms in state CONNECT)
%4|1703095841.652|OFFSET|rdkafka#consumer-1| [thrd:main]: seldon.seldon-mesh.model.tfsimple1.inputs [0]: offset reset (at offset TAIL(1) (leader epoch -1), broker 0) to offset END (leader epoch -1): failed to query logical offset: Local: Host resolution failure
Error: seldon-kafka-0.seldon-kafka-brokers.seldon-mesh.svc:9092/0: Failed to resolve 'seldon-kafka-0.seldon-kafka-brokers.seldon-mesh.svc:9092': nodename nor servname provided, or not known (after 2ms in state CONNECT)
We've debugged the code of seldon cli and discovered the error is returned on this line.
Related bug encountered with the same error: #4776.
Expected behaviour
We were expecting to return the pipeline kafka topics with a JSON format of data outputs of those kafka partitions.
Environment
OS: macos Ventura 13.6.3
Architecture: ARM64
Cloud Provider: Azure
Kubernetes Cluster Version: v5.0.1
Deployed Seldon System Images: seldonv2-controller:2.6.0
The text was updated successfully, but these errors were encountered:
Just a note here: the kafka bootstrap server, which you are pointing to, returns a list of hostnames pointing to the actual kafka brokers. Because those are not visible/accessible from where you are running the seldon cli, the cli cannot connect to them and the error you've described pops up.
In other words, it's not sufficient to simply expose the kafka bootstrap server via a port forward.
I don't think this is a bug per-se, but we'll take it as an improvement request that the seldon pipeline inspect command should work better with the kafka install from within a k8s cluster.
Currently we have installed Seldon Core v2 with version 2.6.0 on a k8s cluster. Based on the installation instructions of kafka on our k8s cluster, we followed the recommendation using Strimzi Operator with all default values from helm chart
strimzi-kafka-operator
(version 0.35.1) and all default values from seldon's helm chartseldon-core-v2-kafka
(version 0.1.0). All control plane and data plane operations are running with the security protocol PLAINTEXT.The goal of this issue is to find out what is the best solution to inspect output data from each model within a Pipeline.
Describe the bug
We built the seldon cli image on our machine and we created a custom configuration on
/home/.config/seldon/cli
with the following:We're port forwarding
seldon-mesh
,seldon-scheduler
andseldon-kafka-bootstrap
services to those ports configured above.We followed the example described here.
We can make an inference through seldon cli, but when we try to do the command
seldon pipeline inspect tfsimples
the following error appears on console:We've debugged the code of seldon cli and discovered the error is returned on this line.
Related bug encountered with the same error: #4776.
Expected behaviour
We were expecting to return the pipeline kafka topics with a JSON format of data outputs of those kafka partitions.
Environment
The text was updated successfully, but these errors were encountered: