How much write QPS can a standalone instance hold?

[pycui]

Can't seem find much relevant and up-to-date data on this. Some numbers assumption here if helpful to answer: Say the standalone instance has no CPU limit and can use up to 128 cores and 500G memory. Say the write request is 1024 dimension with 8K additional text data. Batch 5000 items in a request so each write is close to 64M limit but never exceeded. Is there a way to estimate the write QPS capacity?

Also, what is the behavior if write QPS is larger than capacity? Ideally server should reject excessive requests and it should never fail the milvus instance itself, but I'm seeing in my test that if sending write for too high and too long the Milvus server eventually enters a crashloop, and even write has stopped it seems taking a long time to recover (still crashes sporadically). I'm wondering if this is expected.

[xiaofan-luan]

1. VectorDB is usually not designed for very write intensive task. we would recommend to insert 20-30MB/s at most. having larger topics, more datanodes, and pulsar could help on this.
2. We don't know the config for all the opensource user, so we can not give guideline about what is the right quota limitation. by default if the consumer can not catch up for very long time there is a back pressure. You can setup one your own cluster.
3. For really large amount data, the better way is to use bulkimport , not insertion in large batch.
4. crashlookp may happen for other reasons. maybe your pulsar or minio disk is full, we need logs for details

[pycui]

Unfortunately in my crash loop, there is no log perserved, e.g.k get pod

my-release-etcd-0                               1/1     Running                  0               2d2h
my-release-milvus-standalone-7876f69d74-2875h   0/1     Error                    0               50m
my-release-milvus-standalone-7876f69d74-4c2jw   0/1     Error                    2               5h10m
my-release-milvus-standalone-7876f69d74-4xh5z   0/1     ContainerStatusUnknown   1               39m
my-release-milvus-standalone-7876f69d74-6cx79   0/1     ContainerStatusUnknown   11 (7h5m ago)   9h
my-release-milvus-standalone-7876f69d74-6lwnp   0/1     ContainerStatusUnknown   1               73m
my-release-milvus-standalone-7876f69d74-79hs6   0/1     Error                    0               111m
my-release-milvus-standalone-7876f69d74-8c9vf   0/1     Error                    0               85m
my-release-milvus-standalone-7876f69d74-bdw9x   0/1     ContainerStatusUnknown   1               125m
my-release-milvus-standalone-7876f69d74-bf2qq   0/1     Error                    0               9h
my-release-milvus-standalone-7876f69d74-dlzrz   1/1     Running                  0               7m54s
my-release-milvus-standalone-7876f69d74-fm4cb   0/1     Error                    0               28m
my-release-milvus-standalone-7876f69d74-g5t6p   0/1     ContainerStatusUnknown   1               10h
my-release-milvus-standalone-7876f69d74-gl8mh   0/1     Error                    254             2d2h
my-release-milvus-standalone-7876f69d74-gnxzb   0/1     Error                    0               97m
my-release-milvus-standalone-7876f69d74-h4qvs   0/1     ContainerStatusUnknown   1               3h43m
my-release-milvus-standalone-7876f69d74-h72p5   0/1     Error                    0               9h
my-release-milvus-standalone-7876f69d74-h7sdl   0/1     ContainerStatusUnknown   1               60m
my-release-milvus-standalone-7876f69d74-jrg9c   0/1     ContainerStatusUnknown   3 (5h51m ago)   6h21m
my-release-milvus-standalone-7876f69d74-kp6ss   0/1     ContainerStatusUnknown   1 (3h58m ago)   4h6m
my-release-milvus-standalone-7876f69d74-l99tj   0/1     Error                    0               18m
my-release-milvus-standalone-7876f69d74-lb4px   0/1     ContainerStatusUnknown   4               4h47m
my-release-milvus-standalone-7876f69d74-lh5vd   0/1     Error                    0               3h
my-release-milvus-standalone-7876f69d74-m22dj   0/1     Error                    0               3h14m
my-release-milvus-standalone-7876f69d74-rmgsf   0/1     ContainerStatusUnknown   1               167m
my-release-milvus-standalone-7876f69d74-rtlm6   0/1     ContainerStatusUnknown   1               3h30m
my-release-milvus-standalone-7876f69d74-tdtrx   0/1     ContainerStatusUnknown   4               5h50m
my-release-milvus-standalone-7876f69d74-vvwdl   0/1     Error                    0               153m
my-release-milvus-standalone-7876f69d74-xrvht   0/1     ContainerStatusUnknown   1               139m
my-release-milvus-standalone-7876f69d74-zhkzl   0/1     Error                    2               6h50m
my-release-minio-7867967d7b-tjkq2               1/1     Running                  0               2d2h

Pick any non-running milvus pod, see

✗ k logs my-release-milvus-standalone-7876f69d74-l99tj
unable to retrieve container logs for containerd://32c6caa5f6ea9bafc66d7bf9d878dd2e77faecfbf95f7e43b028bdcdbd42c9ab%              
✗ k logs my-release-milvus-standalone-7876f69d74-l99tj -p
Error from server (BadRequest): previous terminated container "standalone" in pod "my-release-milvus-standalone-7876f69d74-l99tj" not found

[xiaofan-luan]

you need to get the latest log when the node is brought up.
Don't have any idea about why they crash

[pycui]

Unfornatuely the logs were not perserved by k8s.

I'm wondering if this could be the cause of crash loop: I think we were using an in-memory index and the collection definitely grows larger than what the standalone memory can hold. Does Milvus try to resume the collection and the index that was loaded after a restart? If so, this can explain the crash loop, as it will keep crashing due to OOM. Wondering if you can comment on this @xiaofan-luan thanks

[xiaofan-luan]

then it's not gonna to work.
You need enough memory to load all the data before you do anything. otherwise you will hit an oom