Can we add DiskANN in the Milvus size tool?

[pycui]

It's not the index type list currently.

Also, I'm wondering if the DiskANN index will load scalar data into memory. This matters for estimating the memory need for long string fields. Thanks

[yhmo]

We have engineers working on this task. The sizing tool is based on practice experience, which means they will spend lots of time to test.

[yhmo]

DiskANN is a special index for vector field, not for scalar fields.

Index types for vector field:

• in-memory index: IVF_FLAT, IVF_SQ8, IVF_PQ, HNSW, SCANN https://milvus.io/docs/index.md?tab=floating#In-memory-Index
• on-disk(partly) index: DISKANN https://milvus.io/docs/disk_index.md#On-disk-Index
• GPU index: GPU_IVF_FLAT, GPU_IVF_PQ, GPU_CAGRA https://milvus.io/docs/gpu_index.md#GPU-Index

Index types for scalar field: https://milvus.io/docs/index-scalar-fields.md
INVERTED, STL_SORT, TRIE

[pycui]

I know all these indexes. What I meant is there isn't a way to effectively limit the memory footprint of the collection (ideally you would expect Disk-based index to help here, but as you point, it doesn't help scalar fields). I have been testing 1 billion rows with 1024 dim vector + 8K byte scalar fields, and the current features all require a huge amount of memory to be used.

[yhmo]

Enable mmap can help reduce memory usage of scalar fields.
Currently, Milvus doesn't allow users to specify which scalar field to be loaded or unloaded, for some technical reasons. Personally, I think it might be a valuable feature in future versions.

[pycui]

in some k8s environment, mmap can also be undesired when ephemeral storage is limited. Ideally a disk-based index would just use the MinIO storage for a scalar field

[xiaofan-luan]

we do have a mode using minio bat due to the long latency of object storage it could take a minutes to finish the search