How to reduce the memory usage?

[xiaobingxia-at]

Hello Milvus experts, I'm trying to reduce the memory usage while maintaining a relatively high recall rate, I have tried following options:

• Use in-memory index: IVF-SQ8 (IVF-PQ recall rate is a little bit low), the memory usage is about 40% of raw data size.
• Use HNSW + MMAP: The memory usage is 28% of raw data size. (When hosting 160m 1536-d vectors, raw data is 919GB, query node memory usage is 260GB, and Disk Usage is 610GB)
• Use Disk ANN: with default config, the memory usage is 42.6% of raw data size. (When hosting 134.948m 1536-d vectors, raw data is 772.18GB, disk ann index on disk is: 749.5GB, Query Node memory usage is: 319GB). This is concerning, the reason I come to diskann is because of the limited memory resource. If anyway it will take memory which is >40% of raw data, then there is no difference from using in-memory index. Is there anything I did wrong here?

My goal is to reduce the memory usage to be of 10% of raw data size given my data volume. Is there any good advice?
Thanks!

[yhmo]

Typically, the index size compares to the original data size:

• IVF_FLAT/HNSW, a bit larger than the original data size.
• IVF_SQ8/IVF_PQ, 25% ~ 30% of the original data size.
• DISKANN, 1/4 ~ 1/6 of the original data size.

You got 40% memory usage with IVF_SQ8, which is higher than expected. Is your milvus standalone? If so, maybe some compaction/index tasks are not finished.
Normally, if MMAP is enabled, the memory usage is much less than no-mmap. You got 28% for HNSW+MMAP, I think this is expected.
42.6% for DISKANN is also unexpected. Ideally, it should be 20% ~ 25%. Did you change the configuration of PQCodeBugetGBRatio?

[xiaobingxia-at]

PQCodeBugetGBRatio is default value 0.125.

[xiaobingxia-at]

The memory usage is still relatively large given default PGCodeBugetGBRatio value.

[xiaofan-luan]

Hello Milvus experts, I'm trying to reduce the memory usage while maintaining a relatively high recall rate, I have tried following options:

• Use in-memory index: IVF-SQ8 (IVF-PQ recall rate is a little bit low), the memory usage is about 40% of raw data size.
• Use HNSW + MMAP: The memory usage is 28% of raw data size. (When hosting 160m 1536-d vectors, raw data is 919GB, query node memory usage is 260GB, and Disk Usage is 610GB)
• Use Disk ANN: with default config, the memory usage is 42.6% of raw data size. (When hosting 134.948m 1536-d vectors, raw data is 772.18GB, disk ann index on disk is: 749.5GB, Query Node memory usage is: 319GB). This is concerning, the reason I come to diskann is because of the limited memory resource. If anyway it will take memory which is >40% of raw data, then there is no difference from using in-memory index. Is there anything I did wrong here?

My goal is to reduce the memory usage to be of 10% of raw data size given my data volume. Is there any good advice? Thanks!

I don't recommend to use IVFSQ/PQ in your use case.

You can tune diskANN params to reduce the memory foot print(disable the cache and increase the quantazitio ratio)
For HNSW I believe you can insert more data until it reaches OOM.

[xiaobingxia-at]

Thanks @xiaofan-luan ! It is good to know.
Based on the above information, HNSW is doing a lot of random I/O (one data point each time). Does that mean the disk IOPS will have a lot of impact to HNSW performance? I just found that my /proc/sys/fs/aio-max-nr is 65535, instead of 1m. I assume once I change the aio-max-nr to 1m, my HNSW performance will be immediately improved?

[xiaobingxia-at]

Right now, the HNSW search latency is around 300ms - 1s, after stopping the ingestion, and waiting for a few hours, and doing a lot of searches, the latency may reduce to <10ms.
I'm suspecting it is because my ssd disk can only do 65535 aync i/o at the same time.

[liliu-z]

proc/sys/fs/aio-max-nr control the concurrency of AIO which is used by DiskANN. But mmap uses another mechanism to do disk interaction, so this parameter cannot affect the performance of mmap.
For better troubleshooting, checking the IOPS number can help. And plz double check the IOPS capability of the path Milvus is running.

[xiaobingxia-at]

I did the check for "system.io.w_s" and "system.io.r_s" metrics from datadog. Looks like the write request per second is <1000, and read request per second is very very low like <10.

[xiaobingxia-at]

oh, wait, the system.io.r_s was 2800+ if just started the searching process. After ~30 minutes, it gradually went down to 0.