What is the best way to deploy kserve in deployKF?

[edi-bice]

kserve is included as part of most Kubeflow distributions but that does not seem to be the case with deployKF (yet). What is the best way to deploy kserve in deployKF?

[thesuperzapper]

@edi-bice I am just as eager as you to get official KServe support into deployKF, so watch this space!

But until then, you can install standalone KServe using one of their official guides:

• Serverless Mode (Requires: Istio + Knative)
• Raw Mode (Requires: Istio)

You might also want to install ModelMesh Serving (if it's required by your use-case), but note that this component requires you to connect it with an ETCD instance and S3-like object store.

[edi-bice]

Okay thanks, giving it a try but given all the nonstandard names for namespaces and services (mostly due to tacking deploykf) it will take some trial and error to find and rename all.

[thesuperzapper]

@edi-bice can you clarify what you mean by non-standard names?

You might be confusing the standalone KServe with the version that comes with Kubeflow's upstream manifests?

Because the versions that I linked above are for standalone deployments, which can even be deployed if you don't have Kubeflow at all.

[edi-bice]

auth, dashboard, istio-gateway, minio and mysql namespaces are prefixed with deploykf-

I was not confusing the two but was expecting to run into issues given that both Kubeflow (deployKF or otherwise) and KServe, via Knative, (standalone or bundled) require Istio.

I followed the Serverless instructions and as expected it broke my access to the dashboard possibly because of conflicts w.r.t Istio Ingress Gateway etc.

I do have to admit I am still trying to figure out ArgoCD. I was able to deploy deployKF using generator mode, checking in results in git and pointing argocd to it. What do I have to do, if anything, with the KServe deployment I attempted using the Serverless instructions above? Download, add to the project and make sure they're produced as part of the generator output?

[thesuperzapper]

@edi-bice the next version of deployKF 0.1.4 (should be out very soon) plays much nicer with other services exposed on the same Istio Gateway, specifically around things like authentication.

But for now, you can treat KServe as a fully separate app (which is probably a good idea if you are doing critical serving on it), many people even choose to host it on a separate cluster. But even If you use the same cluster as deployKF, you could just use a separate Istio gateway deployment/service (exposed on a slightly different domain) for the KServe stuff.

Note, I am not 100% sure how deployKF will "officially" implement KServe, because there are many trade-offs between stability/performance and consolidation with the other deployKF services, so I am interested in your feedback if you choose one way or the other.

---

Regarding how to include KServe in ArgoCD, you can just install KServe in its own ArgoCD app as a Kustomize-type Application, completely separate from deployKF's apps.

Note, I always recommend using deployKF in "ArgoCD Plugin Mode" because it makes upgrading much easier, and there should not usually be any need to make custom manifests changes (and if there is, we will fix whatever the problem is upstream).

[edi-bice]

Okay, that makes sense. Ideally, KServe is deployed together much like the upstream manifests-based one and the one here https://github.com/GoogleCloudPlatform/kubeflow-distribution which we're familiar with and using for our GCP cloud workloads.

The issues I mentioned earlier could well be due to incompatible versions. I deployed Kubernetes 1.25 a while back and managed to get deployKF running on it. Now, trying to deploy KServe, I wasn't sure if I could get away with the same Istio or install a fresh one. Not many good choices based on the following and indeed net-istio-controller in knative-serving namespace complains that kubernetes v1.25 is not compatible needing at least 1.27 (which Kubeflow isn't tested with AFAIK)

https://kserve.github.io/website/master/admin/serverless/serverless/#recommended-version-matrix

[pbasov]

you could just use a separate Istio gateway deployment/service (exposed on a slightly different domain) for the KServe stuff

@thesuperzapper but then you won't be able to expose the kserve webapp in central dashboard, since it would be a cross-domain iframe, correct?
Answering my own question, webapp works just fine if you serve it behind existing deploykf istio backend.

[dkhachyan]

@pbasov how did you enable Models UI in deployKF dashboard?

[tom-pavz]

Does anyone have an example of how they configured deployKF istio gateway with AWS Load Balancer Controller on EKS using Ingress / Application Load Balancer and then installed Knative and KServe in Serverless Mode? I am having a lot of trouble getting anything other than a 404 on my requests to my inference service from outside the cluster, and I don't understand why.