What features do you want❓

[xiaofan-luan]

Hi All!

Milvus 2.2.X is stable and the search performance improved tremendously.

We're gonna to release 2.2.3 soon, with rolling upgrade and resource group features and bunch of other performance improvement.

Milvus 2.3 will be released by the end of Feb, with enhanced partition functionalities, new velox execution engine, new knowhere 2.0 vector search engine and range search. see detailes at https://wiki.lfaidata.foundation/display/MIL/Feature+plans+for+Milvus+2.3+and+timeline.

Now we are looking for feature list of Milvus 2.4 and Milvus 3.0, pls give us advices.
Some of the possible feature list:

Restful API
SQL support
Graph QL
Cpp/Rust SDK
GPU support
OpenAI/Hugging face/Spark connector
ScaNN index
Multi vector field support
Change data capture
Primary/Backup milvus cluster across dc
Aggregations
Data Iteration
BM25 index
Other datatype support
HDFS storage
PrimaryKey Deduplication
Delete with complicated expression
Bulkload with more data format
Migrate from Milvus 1.0/Faiss/HNSW
Ranking/Reranking
... and anything you are looking for

Thanks

[subrata10xx]

Hi,
I am using the tool available in the below url:
https://milvus.io/tools/sizing/
But trying to understand what is the difference between raw file sixe and memory size. I am trying to get the size of the database/collection that I created.

[yhmo]

"raw file size" is calculated from the vectors count and dimension.
For example, a float vector is 128-dimensional, and each dimension is a float32(4 bytes) value stored in Milvus, both in storage and in CPU memory. The size of this vector is 1284=512 bytes. If there a 1 million vectors stored in Milvus, the total raw file size is 1M512bytes = 512MB.

For different index types, the data size is different. For example, IVF_FLAT index, the data size is nlist512bytes+1M512bytes.

Milvus requires all data to be loaded into CPU memory for the best search performance. If there is no index, it will load the raw data(512MB) into memory. Since Milvus is a complicate system, it has extra overhead. With our experience, we find a ratio to calculate the actual memory cost base on the data size. The "memory size" you saw in the sizing tool is calculate by this ratio and data size.

[Bec-k]

Voting for Aggregations! If we can aggregate unique results after similarity search, that would be super useful.
Like embeddings with these unique labels were found to be the most similar.

So GROUP BY aggregation i think would be the most useful.

What is group by aggregation?
The Group By statement is used to group together any rows of a column with the same value stored in them, based on a function specified in the statement. Generally, these functions are one of the aggregate functions such as MAX() and SUM(). This statement is used with the SELECT command in SQL.

This is second in the list i think, at least for me.
I think that this would introduce a lot more interesting combinations and possibilities into Milvus:
Multi vector field support

SQL and GPU extended support are important as well, but not as first two.

[xiaofan-luan]

We will support count/min/max in 2.3.X and GroupBy/Having will in 2.4 release.

For multi vector，what is a use case you can think of? if it's just store the vectors or we want to run some models on ranking different field search result?

[novemberdude]

How about embedding clustering in real time?

I have a flow of embeddings that arrive in real-time. The following embedding must be compared with ahead embeddings to check if two embeddings can group into a cluster (based on a threshold distance). As the result, we have a collection that contains many clusters, and when a new embedding arrives, it will be assigned to one of those clusters or init a new cluster as a center. I already implemented it with python but that's not efficient when the number of embeddings grows up.

[xiaofan-luan]

Milvus already supported search on streaming datas. Did you mean partition segment by embedding clustering and only search partial segment for better performance?

[Bec-k]

I guess he meant that he wants a filter&decision layer for real-time incoming embeddings, which will distribute embeddings into a different cluster-like or group-like formation. Something like knn in real-time for all incoming embeddings.

I guess that this is out of scope of the Milvus and it is the scope of the application logic.
Milvus can just do a similarity check and additional layer of maps and indexes to forward or add new embedding into group or cluster.

[xiaofan-luan]

I guess he meant that he wants a filter&decision layer for real-time incoming embeddings, which will distribute embeddings into a different cluster-like or group-like formation. Something like knn in real-time for all incoming embeddings.

I guess that this is out of scope of the Milvus and it is the scope of the application logic. Milvus can just do a similarity check and additional layer of maps and indexes to forward or add new embedding into group or cluster.

We are actually thinking of adding some distributed clustering logic, just like spark did. Milvus can be used as fast distance calculation and we need another distributed controller to cluster data cost more than one single machine.
Do you think it will be helpful?

[Bec-k]

If that will help to arrange groups based on some statements (like similarity score), then that would help to implement or move various implementations to Milvus and thus scale it massively.
One of such implementations is Deepsort, which is in its base very short-lived, but could be drastically expanded with Milvus grouping functionality.

[filip-halt]

Another key feature that I think shoudl be worked on is the ability to increase replication factor on the go. As a system grows it makes sense to be able to increase the replication without having to shutdown the system to release and reload.

[xiaofan-luan]

I thought that would be change In memory replica count on flight. We are actually working on that

[filip-halt]

One more feature that might be very useful is the ability to only search indexed segments. Many users like having the most up to date data as possible without bogging down the system. Currently the only way to get around this is hack-y and not guaranteed to work.

[congqixia]

I think this pr completes part of your proposal. #21994
we could add more search options to ignore non-indexed segments as well.

[dee-corr]

One feature that would be useful is the ability to search multi-vector in one search

[xiaofan-luan]

some data modalities (like audio) need multiple vectors for segments of the same source. on search, you may need to pool the results of vectors from the same source

does all the vectors generated from the same list?
Should be something like a vector list?
What is the typical list size from the typical use case?

[cyrusvahidi]

For my use case, the list size is 5-20 elements. Given ordered search results, each result should be aggregated with all other results that share a property (i.e. same source). Each vector from a source is identified by its index in the vector list.

[xiaofan-luan]

For my use case, the list size is 5-20 elements. Given ordered search results, each result should be aggregated with all other results that share a property (i.e. same source). Each vector from a source is identified by its index in the vector list.

That is under the plan! Embedding list (less than 500 elements) will be part of 2.3.x

[cyrusvahidi]

great! what's the best approach to hacking something together to do this now? i.e aggregate the IP scores of a whole vector list

[yiwen92]

@dee-corr @filip-halt "A cool use case is using different embeddings for a single source and doing weighted search on multiple vectors." So in your case, are you trying to separate these 2 embedding vectors into 2 collections or combine them into one?

[Ivana-DS]

I would like to have folowing features:

1. List as a data type: [Feature]: Supporting List of Strings and List of Integers as data type #22738
2. Supporting the search in strings with %string% not only for prefix like in the moment: prefix%

[xiaofan-luan]

I would like to have folowing features:

1. List as a data type: [Feature]: Supporting List of Strings and List of Integers as data type #22738
2. Supporting the search in strings with %string% not only for prefix like in the moment: prefix%

1. already supported with json.
2. already on the roadmap

[NehaTamore-Draup]

sparse-dense hybrid search would be helpful too!

[xiaofan-luan]

sparse-dense hybrid search would be helpful too!

Already on milestone!

[Bec-k]

Would be cool to have a functionality to automatically group vectors by similarities. We could have a specific type of collections for that. For example:

• app inserts a batch of new vectors
• Milvus is assigning each of those vectors to an existing group of vectors by similarity
• else creates new group if match is below required minimal similarity threshold
• single vector can be assigned only to a single group (Hungarian algorithm / Munkres assignment algorithm)
• each group of vectors should have TTL or vector should have TTL

That way we can have different app implementations at scale. I have described DeepSORT internals. If we could move DeepSORT logic into Milvus and make YOLO detections and features extraction services distributed/stateless, that would be the next step for our company.

[jiaoew1991]

I think it would be more flexible to leave the decision of whether to group or not to the user-side code. I suggest following this process: search topK -> determine if a new group should be created -> insert. At this point, the groupID can be used as a regular scalar field. 🤔

[Bec-k]

Then we need batch operations for both. Imagine how much insert and search operations we would need to perform. This won't fit into real-time video processing. I'm pretty sure about that.

[Bec-k]

Imagine you have 10 YOLO detections in a video frame, scenario is as follows:

• Send 10 Milvus similarity requests
• wait for responses...
• Analyze 10 responses and send create segment/group request or create collection requests
• wait for responses...
• use returned collection/group/segment IDs to assign them to local vectors

With my above proposed feature description, it would look:

• send 1 batch insert
• wait for response...
• use returned collection/group/segment IDs to assign them to local vectors

[Bec-k]

As you can see, there are much lesser steps required to perform what we need and less data round-trips required as well. Add network latencies here as well.

[Bec-k]

This will only work well if standalone Milvus version will be used and our app will be deployed on the same machine where Milvus is installed. Which kills the distributed and scalable features Milvus provides.

[ruixiu]

Support groupby + top

[xiaofan-luan]

GroupBy feature is already in the plan

[Nicolas2912]

Would it be beneficial or even practical to incorporate an integrated text embedding functionality within Milvus? The essence of this feature would be to automatically translate any inserted text into its corresponding vector representation using a pre-selected embedding technique. Following this transformation, a search query composed of text could result in a return of the 'n' most similar texts, based on their proximity in the vector space.

This idea sounds appealing as it could streamline the process of text-to-vector conversion, and improve the overall efficiency of the text search within Milvus. However, it raises several considerations. How would different embedding methods affect the quality and relevancy of search results? How could this feature accommodate the broad range of languages and semantic complexities in text data? Furthermore, embedding processes can be computationally heavy; could the real-time transformation of text into vectors impact system performance?

While the concept presents a potential advancement in handling text data within Milvus, it warrants thorough exploration of these issues to determine its practicality and potential benefits.

[xiaofan-luan]

Would it be beneficial or even practical to incorporate an integrated text embedding functionality within Milvus? The essence of this feature would be to automatically translate any inserted text into its corresponding vector representation using a pre-selected embedding technique. Following this transformation, a search query composed of text could result in a return of the 'n' most similar texts, based on their proximity in the vector space.

This idea sounds appealing as it could streamline the process of text-to-vector conversion, and improve the overall efficiency of the text search within Milvus. However, it raises several considerations. How would different embedding methods affect the quality and relevancy of search results? How could this feature accommodate the broad range of languages and semantic complexities in text data? Furthermore, embedding processes can be computationally heavy; could the real-time transformation of text into vectors impact system performance?

While the concept presents a potential advancement in handling text data within Milvus, it warrants thorough exploration of these issues to determine its practicality and potential benefits.

Good suggestion! We actually open sourced another project towhee, for embedding generation https://github.com/towhee-io/towhee

And yes, I'm thinking of how to integrate embedding and reranking model into Milvus

[xiaofan-luan]

maybe something like a plugin pod.
Any suggestion would be recommend

[congguosn]

Hi I think ipv6 is definitely a trend to overcome the shortage of ipv4 addresses, especially when using cloud service. Do the community has plan to support it? I think the ROI should be high.

BTW, aws is about to charge for ipv4 public addresses from Feb. 2024
https://www.infoq.com/news/2023/08/aws-ec2-public-ipv4

[xiaofan-luan]

Agreed, this could something in our roadmap

[490626721]

I hope to improve the restful related functions, align them with other SDK interface functions, and make it convenient for other languages to call related interfaces。

[xiaofan-luan]

Anything else you want to use is missing?

[xiaofan-luan]

And also any contribution is welcomed

[490626721]

The RESTful documentation is very perfect, but it seems to lack the management indexes related interface. Additionally, the Create Collection API cannot freely customize each field like pySDK, and can set the relevant information for each field

[xiaofan-luan]

So far, for the ease of usage, we only support certain kind of index.
Yes, we do have plan to improve the restful, please open issues on features you want to use and we will keep updating

[ShravanSunder]

@xiaofan-luan +1 for full parity for create collection and fields with restful

[Jkintree2]

Milvus is integrated into huggingface. If huggingface spaces does not provide a way to share vector databases for retrieval augmentation, some way of having a hub for vector databases would be helpful.

[Jkintree2]

If vectors representing arguments that are made in conversations and debates are stored in a vector database, we might need a higher layer of continual training of a model that learns the strongest weights between vectors that are best supported by reason and evidence.

[Jkintree2]

I just watched "How to add conversational memory to an LLM using langchain : vector database fundamentals" in the zilliz channel at YouTube. 

It would be nice to have a milvus lite that runs on smartphones to store vector representations of conversational history on device. 

[ShravanSunder]

@xiaofan-luan restful typescript client library. support for milvus and zilliz.

And feature parity for restful api

[ShravanSunder]

@shanghaikid thanks for response and workaround. In genearl grpc doesn't work really well with edge or many serverless implementations. It would be greatly appreciated to have a fleshed out http client library in typescript.

[shanghaikid]

You are right, definitely I will add a http client in the node sdk. but it takes time. stay tuned.
I also create a nextjs example https://github.com/milvus-io/milvus-sdk-node/tree/main/examples/nextjs .

[shanghaikid]

@ShravanSunder typescript http client pr is under review. it will be released soon.

[ShravanSunder]

@shanghaikid thanks for the update! I checked out the PR. It looks great. The one feedback i had is that axios is not compatable with cloudflare workers, vercel edge functions. It would be better to use fetch.

milvus/HttpClient.ts https://github.com/milvus-io/milvus-sdk-node/blob/b72797f98d4720dcd1d8bd4d3e5d53887241cd39/milvus/HttpClient.ts

[shanghaikid]

ok, I will update it.

[kish5432]

Topic: Access Control Request for 'rbac_user' User in Milvus Vector Database
Details:
I am writing to request a database level access control configuration for a user in our Milvus Vector Database.

Example Sceanario:

1. use root user to create a database 'rbac_db'. and create a collection in the default db
2. use root user to login to 'rbac_db' database, create a user 'rbac_user' and give the below privilages.
   role.grant("Collection", "", "", db_name='rbac_db')
3. Create a collection in 'rbac_db'.
4. use the user 'rbac_user' to login to the default db, list the collections. It should not allow to login to default database as 'rbac_user' dont have privileges.
5. use the user 'rbac_user' to login to 'rbac_db', list the collections. Here it should show the collections as its have the privileges on 'rbac_db' database.

The 'rbac_user' user should not be allowed to log in to the other databases like 'default' database.
Collections from the 'default' database should not be visible or accessible to the 'rbac_user' user.

This access control is essential for maintaining the security and privacy of our database. Please ensure that the necessary configurations are in place to restrict 'rbac_user' from accessing the 'default' database and its associated collections.

Thanks
Kishore

[xiaofan-luan]

@SimFG
any advice?

[SimFG]

@xiaofan-luan @kish5432
The current security verification method used by Milvus is Basic Authentication. This means that user and permission verification are performed on each request, eliminating the need for a separate login step. However, if we were to implement an authentication method like JWT, a login step would be required. With JWT, users would need to log in and obtain a corresponding token before making any requests. Although this approach offers increased security, it also introduces additional overhead.

Additionally, regarding the "list collections" interface, disabling it would have consequences. Even if a user has been granted collection permissions (e.g., create collection), they would be unable to utilize this interface. This is because the SDK relies on certain global-level permission operations to enhance the convenience of SDK usage. Considering this aspect, new users are granted some permissions from the public role by default. For more details, please refer to the following documentation: Users and Roles Milvus.

[ShravanSunder]

@xiaofan-luan in general would be very useful to have ingestion streams and output streams out of Milvus/zilliz

Specifically, it would be great to have the zilliz confluent Kafka stream work with any kakfka instance. i’d like to use it with upstash serverless kafka https://upstash.com/docs/kafka/overall/get-started

[xiaofan-luan]

For opensource kafka users, you may want to try airbyte?
And we are working with flink CDC team

[ShravanSunder]

@xiaofan-luan Hmm, it would be another component to manage, it would be much easier to directly use a any kafka sink with milvus.
my usecase

• i have data writes to postgres (supabase)
• I want to use debezium and push those changes to upstash kafka. The reason i want to use upstash is that they good pricing per usage (no fixed costs) for kafka.
• I want to use the upstash kafka as a sink to feed information into milvus

[xiaofan-luan]

did upstash support kafka java API?
we are thinking of supporting kafka connector but maybe not it's restful API

[ShravanSunder]

@xiaofan-luan I did some checking. Upstash supports native kafka clients (TCP) and Rest API over http.

kafka api

The native api supports java / go / nodejs / python
They have examples of java api here: https://upstash.com/docs/kafka/overall/kafkaapi#create-a-topic

[xiaofan-luan]

Native kafka API should work.

[TommyLemon]

SQL & JDBC/ODBC library(Java is preferred)

[TommyLemon]

I wrote a plugin to adapt Milvus and Tencent/APIJSON
https://github.com/APIJSON/apijson-milvus

BTW, official JDBC/ODBC library is still a better way for more user scenarios

[MR-GOYAL]

I am trying to deploy Milvus db on IPV6 only Kubernetes cluster and getting some error ? Can Milvus db have IPV6 support in next release?

[xiaofan-luan]

@LoveEachDay
any comments?

[LMERose]

Hello, I would like to ask if milvus provides an LRU（Least Recently Used） method? How to use it?

[yiwen92]

LRU is in plan, but we would like to make it automatically in tired-storage arch, what's your scenario and requirement?

[vgonisanz]

It may be interesting to have:

1. Datatype support for Dates, Is seems already requested in this issue. Our current approach is to store as INT64 but require some conversions in client side.
2. A tool to migrate schemas in already created collections. SQLAlchemy support Dialects with several DDBBs to migrate schemas. Alembic is an example how to deal with a production environment when your schemas change along time. It is possible to do it manually thought pymilvus but require manual scripts.

[rose-jinyang]

Hello @xiaofan-luan
How are you?
I want to increase the limit of the number of the most similar result to return.

How can I do it?

[vgonisanz]

As far as I know, using Python using the milvus client, it is not possible to perform search/queries above that limit, but using the ORM query_iterator section it is possible.

Does anyone know what causes this discrepancy? It is very useful to use the object-oriented client and it is a bit strange to implement only some functionalities with ORM, which requires handling connections with aliases properly.

[rose-jinyang]

Thank you @yhmo

[yhmo]

@vgonisanz
We have noticed this discrepancy, we will add iterator methods to MilvusClient.
#33143

[yiwen92]

@rose-jinyang Just wondering how much in your case you want to increase to?

[douxiaofeng99]

support colbertv2 for retrieval and ranking.

[patrickthoreson]

@xiaofan-luan last year you mentioned "We will support count/min/max in 2.3.X and GroupBy/Having will in 2.4 release.";
however the count() feature works only in scalar query/collection.query() and group_by_field="somestring" only works for vector search/collection.search().
Is there a way to combine these two concepts (count() and group_by), like we do in SQL?
(in SQL we're able to combine the two concepts in the same SELECT command, allowing us to obtain a count(*) for each distinct value of specified in the GROUP BY clause)

[xiaofan-luan]

Agreed, this is exactly we want to offer. @MrPresent-Han is actively working on it

[r0x07k]

Hi,

It'd be good to have indexing of JSON fields.

[xiaofan-luan]

what kind of index do you want? what kind of json do you have?

[r0x07k]

Currently, it's just a set of simple key-value pairs.

'metadata': {'absolute_directory_path': 'c:\\Temp\\Test',
   'index': '19',
   'size': '20',
   'file_name': 'lorem.txt'}}

For example, we might need to look up a file name.

I think a B-tree would work.

[xiaofan-luan]

So ideally you do filter like absolute_directory_path=='c:\Temp\Test' && file_name == 'lorem.txt'?

do we need to build index on size and index as well? basically I want to know how dynamic the schema can be

[r0x07k]

Yes, this is correct by path and name. It's unlikely that index and size will ever be needed, but our app is in active development, so new fields may be added soon. At this point, we're not planning to store nested JSON objects as values—only strings and numbers.

Our goal at this point is to avoid storing any collections' metadata in a separate DB and utilize Milvus itself.

[xiaofan-luan]

Understand, we've been investigating this feature for a well.
With tantivy index this seems to be something we can do (Without nested json data structre).
I will create an issue for that feature. Please keep track

[shabeerak]

We’re in the process of migrating from Solr to Milvus, but we’re currently stuck due to a limitation involving array filtering. This is critical for one of our key features. Specifically, we’re looking for an ARRAY_SUBSET_OF filter, which would be the opposite of ARRAY_CONTAINS_ALL.

Here’s how the filters would work:

• filter="ARRAY_CONTAINS_ALL(color_coord, [7, 8])": This would match all entities where color_coord includes both 7 and 8.
• filter="ARRAY_SUBSET_OF(color_coord, [7, 8, 9])": This would match entities where color_coord is a subset of [7, 8, 9], such as [7], [7, 8], or [9, 7], but not [7, 8, 9, 10] or [7, 10].

Wondering if this is something that can be added as a feature

[liliu-z]

ARRAY_CONTAINS_ALL has already been supported https://milvus.io/docs/array_data_type.md#Advanced-filtering

[shabeerak]

ARRAY_CONTAINS_ALL has already been supported https://milvus.io/docs/array_data_type.md#Advanced-filtering

thanks @liliu-z, I did notice ARRAY_CONTAINS_ALL was implemented. I was just comparing the new requested feature ARRAY_SUBSET_OF with existing ARRAY_CONTAINS_ALL for a better illustration.

In Milvus 2.5 and the master branch, we have introduced a mechanism that facilitates developers in adding custom filter functions. Refer to commit 3628593

thanks @czs007, this will be helpful. any dates planned for 2.5 release yet? also just want to check if there is any documentation on how to implement/add custom filters?

[liliu-z]

2.5 is about to release these two weeks.

[czs007]

@shabeerak You're welcome.

The current design document is overly rudimentary. In the next couple of days, I intend to prepare a more detailed development tutorial (to be placed in the Milvus repository), using starts_with as an example to demonstrate where code modifications and additions are required.

Furthermore, I have opened a dedicated issue #37607 where we can thoroughly discuss the technical details.

[shabeerak]

@czs007 I was wondering if u got a chance to have some documentation on this