How do I connect to a milvus server ignoring ssl?

[matsuo1234567]

What I'm trying to do

I want to connect to milvus ignoring SSL.

Status

Environment

• Milvus version: 2.4.9
• Deployment mode(standalone or cluster): cluster
• MQ type(rocksmq, pulsar or kafka): pulsar
• SDK version(e.g. pymilvus v2.0.0rc2): pymilvus v2.4.6
• OS(Ubuntu or CentOS): macOS Sonoma14.6.1
• CPU/Memory:
• GPU:
• Others: use GKE cluster

Python code

hello_milvus.py

• Change connections.connect(host="34.160.94.83", port="80", secure=False, server_name="my-release.milvus.io", timeout=10)

Execution result

% python hello_milvus.py                     

=== start connecting to Milvus     ===

[__internal_register] retry:4, cost: 0.27s, reason: <_InactiveRpcError: StatusCode.UNAVAILABLE, Socket closed>
[__internal_register] retry:5, cost: 0.81s, reason: <_InactiveRpcError: StatusCode.UNAVAILABLE, Socket closed>
[__internal_register] retry:6, cost: 2.43s, reason: <_InactiveRpcError: StatusCode.UNAVAILABLE, Socket closed>
[__internal_register] retry:7, cost: 3.00s, reason: <_InactiveRpcError: StatusCode.UNAVAILABLE, Socket closed>
[__internal_register] retry:8, cost: 3.00s, reason: <_InactiveRpcError: StatusCode.UNAVAILABLE, Socket closed>
[__internal_register] retry:9, cost: 3.00s, reason: <_InactiveRpcError: StatusCode.UNAVAILABLE, Socket closed>
RPC error: [__internal_register], <MilvusException: (code=<bound method _InactiveRpcError.code of <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Socket closed"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Socket closed", grpc_status:14, created_time:"2024-09-04T11:15:59.839503+09:00"}"
>>, message=Retry timeout: 10s, message=Socket closed)>, <Time:{'RPC start': '2024-09-04 11:15:46.885804', 'RPC error': '2024-09-04 11:15:59.841022'}>
Traceback (most recent call last):
  File "/Users/matsuo/dev/milvus-distributed/hello_milvus.py", line 32, in <module>
    connections.connect(host="34.160.94.83", port="80", secure=False, server_name="my-release.milvus.io", timeout=10)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/orm/connections.py", line 447, in connect
    connect_milvus(**kwargs, user=user, password=password, token=token, db_name=db_name)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/orm/connections.py", line 398, in connect_milvus
    gh._wait_for_channel_ready(timeout=timeout)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 155, in _wait_for_channel_ready
    raise e from e
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 148, in _wait_for_channel_ready
    self._setup_identifier_interceptor(self._user, timeout=timeout)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 266, in _setup_identifier_interceptor
    self._identifier = self.__internal_register(user, host, timeout=timeout)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/decorators.py", line 147, in handler
    raise e from e
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/decorators.py", line 143, in handler
    return func(*args, **kwargs)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/decorators.py", line 182, in handler
    return func(self, *args, **kwargs)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/decorators.py", line 93, in handler
    raise MilvusException(e.code, f"{to_msg}, message={e.details()}") from e
pymilvus.exceptions.MilvusException: <MilvusException: (code=<bound method _InactiveRpcError.code of <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Socket closed"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Socket closed", grpc_status:14, created_time:"2024-09-04T11:15:59.839503+09:00"}"
>>, message=Retry timeout: 10s, message=Socket closed)>

Additional information

% kubectl get ingress my-release-milvus -o yaml   

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.gcp.kubernetes.io/pre-shared-cert: mcrt-1264c5a6-d68d-437d-9678-4cc86ed27db9
    ingress.kubernetes.io/backends: '{"k8s-be-31258--8a08b0129563e43a":"UNHEALTHY","k8s1-8a08b012-default-my-release-milvus-19530-1a635530":"UNHEALTHY"}'
    ingress.kubernetes.io/forwarding-rule: k8s2-fr-dqg55urx-default-my-release-milvus-hf0x9kro
    ingress.kubernetes.io/https-forwarding-rule: k8s2-fs-dqg55urx-default-my-release-milvus-hf0x9kro
    ingress.kubernetes.io/https-target-proxy: k8s2-ts-dqg55urx-default-my-release-milvus-hf0x9kro
    ingress.kubernetes.io/ssl-cert: mcrt-1264c5a6-d68d-437d-9678-4cc86ed27db9,k8s2-cr-dqg55urx-825r3uwvq56p2xuq-4fba4ea797354060
    ingress.kubernetes.io/static-ip: k8s2-fr-dqg55urx-default-my-release-milvus-hf0x9kro
    ingress.kubernetes.io/target-proxy: k8s2-tp-dqg55urx-default-my-release-milvus-hf0x9kro
    ingress.kubernetes.io/url-map: k8s2-um-dqg55urx-default-my-release-milvus-hf0x9kro
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"networking.gke.io/managed-certificates":"my-release-milvus-tls"},"name":"my-release-milvus","namespace":"default"},"spec":{"rules":[{"host":"my-release.milvus.io","http":{"paths":[{"backend":{"service":{"name":"my-release-milvus","port":{"number":19530}}},"path":"/","pathType":"Prefix"}]}}],"tls":[{"hosts":["my-release.milvus.io"],"secretName":"my-release-milvus-tls"}]}}
    networking.gke.io/managed-certificates: my-release-milvus-tls
  creationTimestamp: "2024-09-03T05:31:28Z"
  finalizers:
  - networking.gke.io/ingress-finalizer-V2
  generation: 2
  name: my-release-milvus
  namespace: default
  resourceVersion: "11899447"
  uid: 8a74d529-51e9-4fb6-b164-2f8b993fe674
spec:
  rules:
  - host: my-release.milvus.io
    http:
      paths:
      - backend:
          service:
            name: my-release-milvus
            port:
              number: 19530
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - my-release.milvus.io
    secretName: my-release-milvus-tls
status:
  loadBalancer:
    ingress:
    - ip: 34.160.94.83

% kubectl get svc my-release-milvus
NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
my-release-milvus   ClusterIP   10.104.68.162   <none>        19530/TCP,9091/TCP   20h

% kubectl get pods -l app.kubernetes.io/instance=my-release

NAME                                           READY   STATUS    RESTARTS      AGE
my-release-etcd-0                              1/1     Running   0             22h
my-release-etcd-1                              1/1     Running   0             22h
my-release-etcd-2                              1/1     Running   0             22h
my-release-milvus-datanode-d9d84b68f-jgj7m     1/1     Running   1 (22h ago)   22h
my-release-milvus-indexnode-697bcc9778-w6hvf   1/1     Running   1 (22h ago)   22h
my-release-milvus-mixcoord-7fc4fc4dbc-w2n2j    1/1     Running   0             20h
my-release-milvus-proxy-778844bf45-fc9wt       1/1     Running   1 (22h ago)   22h
my-release-milvus-querynode-7d66fdc5bd-29226   1/1     Running   0             22h

% curl -v http://34.160.94.83:80

*   Trying 34.160.94.83:80...
* Connected to 34.160.94.83 (34.160.94.83) port 80
> GET / HTTP/1.1
> Host: 34.160.94.83
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 502 Bad Gateway
< Content-Type: text/html; charset=UTF-8
< Referrer-Policy: no-referrer
< Content-Length: 332
< Date: Wed, 04 Sep 2024 04:17:22 GMT
< 

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>
* Connection #0 to host 34.160.94.83 left intact

% nc -zv 34.160.94.83 80
Connection to 34.160.94.83 port 80 [tcp/http] succeeded!

% nslookup my-release.milvus.io

Server:         10.0.63.3
Address:        10.0.63.3#53

** server can't find my-release.milvus.io: NXDOMAIN

% pip show pymilvus

Name: pymilvus
Version: 2.4.3
Summary: Python Sdk for Milvus
Home-page: 
Author: 
Author-email: Milvus Team <milvus-team@zilliz.com>
License: 
Location: /Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages
Requires: environs, grpcio, milvus-lite, pandas, protobuf, setuptools, ujson
Required-by: milvus_cli

my-release-milvus-service.yml

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/app-protocols: '{"milvus":"HTTP2"}'
    cloud.google.com/backend-config: '{"default": "my-release-backendconfig"}'
    cloud.google.com/neg: '{"ingress": true}'
    cloud.google.com/neg-status: '{"network_endpoint_groups":{"19530":"k8s1-8a08b012-default-my-release-milvus-19530-1a635530"},"zones":["asia-northeast1-a"]}'
    meta.helm.sh/release-name: my-release
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2024-08-30T07:39:21Z"
  labels:
    app.kubernetes.io/instance: my-release
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: milvus
    app.kubernetes.io/version: 2.4.10
    component: proxy
    helm.sh/chart: milvus-4.2.9
  name: my-release-milvus
  namespace: default
  resourceVersion: "11824268"
  uid: 15404913-e5e5-4c1d-87ad-a39f0443cf6d
spec:
  type: LoadBalancer
  selector:
    app: my-release-milvus
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: milvus
    port: 19530
    protocol: TCP
    targetPort: milvus
  - name: metrics
    port: 9091
    protocol: TCP
    targetPort: metrics
  selector:
    app.kubernetes.io/instance: my-release
    app.kubernetes.io/name: milvus
    component: proxy
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Concerns

Is external storage for GCP necessary? If so, please tell me why.

[yhmo]

To enable SSL for milvus:

1. generate some certificate files by following the steps in this document: https://milvus.io/docs/tls.md
2. config the milvus server
   If the milvus is deployed by helm, read this doc to know how to configure it: https://milvus.io/docs/configure-helm.md?tab=component
   Basically, configurations are defined in the value.yaml and passed to the milvus.yaml for milvus:

extraConfigFiles:
  user.yaml: |+
    tls:
      serverPemPath: /milvus/tls/server.pem
      serverKeyPath: /milvus/tls/server.key
      caPemPath: /milvus/tls/ca.pem
    
    common:
      security:
        tlsMode: 2

Set tlsMode to 1 for one-way authentication. Set tlsMode to 2 for two-way authentication.

The "/milvus/tls" is an internal virtual path inside pod, which should be mapped to the outside path. Anyway, the milvus proxy must access these certificate files to build ssl connection.

3. Use SDK to connect the server
   By pymilvus, you can copy the certificate files into the local path of your client, and connect the milvus proxy address like this:

milvus_client = MilvusClient(uri=URI,
                            client_pem_path="/my_local_path/tls/client.pem",
                            client_key_path="/my_local_path/tls/client.key",
                            ca_pem_path="/my_local_path/tls/ca.pem",
                            server_name='localhost')
print(milvus_client.list_collections())

If the tlsMode is 2, the client must provide client.pem, client.key and ca.pem for the connection.

[xiaofan-luan]

great answer!

[yhmo]

Storage service is optional, you can choose GCS, S3, MinIO, etc.
https://milvus.io/docs/deploy_s3.md
https://milvus.io/docs/gcp.md#Use-Google-Cloud-Storage-GCS-as-external-object-storage
https://milvus.io/docs/object_storage_operator.md#Use-Amazon-Web-Service-AWS-S3-as-external-object-storage

[matsuo1234567]

Thank you, @yhmo

Since I am using Helm, I also consulted https://milvus.io/docs/configure-helm.md?tab=component.
I still cannot connect.

I can connect to milvus without storage, right?

% kubectl get svc
NAME                          TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                               AGE
kubernetes                    ClusterIP      10.104.64.1     <none>          443/TCP                               4d22h
my-release-etcd               ClusterIP      10.104.64.171   <none>          2379/TCP,2380/TCP                     2d1h
my-release-etcd-headless      ClusterIP      None            <none>          2379/TCP,2380/TCP                     2d1h
my-release-milvus             LoadBalancer   10.104.68.162   35.243.101.14   19530:31365/TCP,9091:32239/TCP        22h
my-release-milvus-datanode    ClusterIP      None            <none>          9091/TCP                              4d22h
my-release-milvus-indexnode   ClusterIP      None            <none>          9091/TCP                              4d22h
my-release-milvus-mixcoord    ClusterIP      10.104.68.2     <none>          9091/TCP                              4d22h
my-release-milvus-querynode   ClusterIP      None            <none>          9091/TCP                              4d22h
my-release-minio              ClusterIP      10.104.65.163   <none>          9000/TCP                              2d1h
my-release-minio-svc          ClusterIP      None            <none>          9000/TCP                              2d1h
my-release-pulsar-bookie      ClusterIP      None            <none>          3181/TCP,8000/TCP                     4d22h
my-release-pulsar-broker      ClusterIP      None            <none>          8080/TCP,6650/TCP                     4d22h
my-release-pulsar-proxy       ClusterIP      10.104.66.214   <none>          8080/TCP,6650/TCP                     4d22h
my-release-pulsar-recovery    ClusterIP      None            <none>          8000/TCP                              4d22h
my-release-pulsar-zookeeper   ClusterIP      None            <none>          8000/TCP,2888/TCP,3888/TCP,2181/TCP   4d22h

% telnet 35.243.101.14 443 

Trying 35.243.101.14...
Connected to 14.101.243.35.bc.googleusercontent.com.


% nc -zv 35.243.101.14 443 

Connection to 35.243.101.14 port 443 [tcp/https] succeeded!

hello_milvus.py
Change code this
connections.connect(host="35.243.101.14", port="443", secure=True, server_name="my-release.milvus.io", timeout=30)

% python hello_milvus.py   

=== start connecting to Milvus     ===

Traceback (most recent call last):
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 147, in _wait_for_channel_ready
    grpc.channel_ready_future(self._channel).result(timeout=timeout)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/grpc/_utilities.py", line 162, in result
    self._block(timeout)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/grpc/_utilities.py", line 106, in _block
    raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/matsuo/dev/milvus-distributed/hello_milvus.py", line 32, in <module>
    connections.connect(host="35.243.101.14", port="443", secure=True, server_name="my-release.milvus.io", timeout=30)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/orm/connections.py", line 447, in connect
    connect_milvus(**kwargs, user=user, password=password, token=token, db_name=db_name)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/orm/connections.py", line 398, in connect_milvus
    gh._wait_for_channel_ready(timeout=timeout)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 150, in _wait_for_channel_ready
    raise MilvusException(
pymilvus.exceptions.MilvusException: <MilvusException: (code=2, message=Fail connecting to server on 35.243.101.14:443, illegal connection params or server unavailable)>