Clustering Vectors to Groups

[OrjwanZaafarani]

Does Milvus support clustering vectors into groups where each group contains similar vectors?

[yhmo]

Milvus is a vector similarity search service, the underlying engine is FAISS and HNSW lib. The FAISS provides IVF index, k-means clustering algorithm, this kind of index can cluster vectors into groups where each group contains similar vectors.

[mihow]

This is great news! Is it possible to link to an example of how to do k-means clustering with Milvus from a practical level? Is it possible to use the FAISS k-means implementation without loading all of the vectors into memory?

[yhmo]

This is great news! Is it possible to link to an example of how to do k-means clustering with Milvus from a practical level? Is it possible to use the FAISS k-means implementation without loading all of the vectors into memory?

Milvus encapsulates faiss as a lib, and expose create_index() API to user to build k-means clusters. But the k-means clusters are invisible to user.
In python SDK, the create_index():

        collection = Collection(name=COLLECTION_NAME)

        index_param = {
            "index_type": "IVF_FLAT",
            "params": {"nlist": 1024},
            "metric_type": "L2"}
        collection.create_index(VECTOR_FIELD_NAME, index_param)

The create_index() doesn't require data loaded into memory.

[xiaofan-luan]

This is great news! Is it possible to link to an example of how to do k-means clustering with Milvus from a practical level? Is it possible to use the FAISS k-means implementation without loading all of the vectors into memory?

Milvus encapsulates faiss as a lib, and expose create_index() API to user to build k-means clusters. But the k-means clusters are invisible to user. In python SDK, the create_index():

        collection = Collection(name=COLLECTION_NAME)

        index_param = {
            "index_type": "IVF_FLAT",
            "params": {"nlist": 1024},
            "metric_type": "L2"}
        collection.create_index(VECTOR_FIELD_NAME, index_param)

The create_index() doesn't require data loaded into memory.

I guess if the user want to do clustering across the whole collection the answer is no. Milvus only supports segment level clustering

[mistborn17]

Is it possible to extract the centroids of these clusters ?

[xiaofan-luan]

Is it possible to extract the centroids of these clusters ?

right now the answer is no.
and this is also only centroids for one segment, not the entire collection