Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal for vector db semantic convention #1231
base: main
Are you sure you want to change the base?
Proposal for vector db semantic convention #1231
Changes from 3 commits
fa8ee3070687205e12a863b61784828bacc53d82d4a3330ffe5ff387da6649bd99ec1061b0f2cceae2ca6db7ec5e5e0d9dae0e06603b67bf93d2cbef411554daa0a14bc8a63ca5f8661d996cd9a10e75ffd0f2e71c6bd009feb74d235776681dca47ff03da1523bcb9fd891f6bc2ddb1fc90f3fFile filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
suggesting more explicit name like
db.vector.search.similarity_metricThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ezimuel I think you also missed this change, is marked as resolved without any updates :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in a3330ff
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW, who/how/when is going to populate it?
It seems it's only available at index creation time and not available at query or insert time. I.e. very rarely.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct, this is generally provided during the index creation but some databases uses it also in the query (e.g. Qdrant).
I think we can also leverage the Conditionally Required level for some attributes that are not always available, like the
db.vector.search.similarity_metric, WDYT?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lmolkova just a reminder for this, thanks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, it makes sense!
BTW I think the attribute should be
db.search.similarity_metricor similar - similarity is not always based on vectors and we don't need to limit this attribute.Check failure on line 556 in model/registry/db.yaml
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need both - the name and the id?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
idis the identifier of the vector, thenameis the field name that contains the vector. Many database uses both but some uses onlynameof the field. Maybe we should use a better naming here.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lmolkova what do you think about this? Thanks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you provide some examples of id and name?
Let's pick a few (2-3) popular databases and explain what id and name would mean in their context.
E.g. I look into MongoDB and I don't understand what id or name would be.
Or I look into Azure Search and don't understand what should go into id.
I look into pinecone and it does not talk about ids.
Also, perhaps by vector you mean index? Or if it's about individual vectors, then how the index would be represented?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lmolkova Here some examples:
pathin search here andfield-namein the definition here)fieldname_embedding, here)By db.vector.name I mean the field name in the document that will be used for embedding (most common for DB like MongoDB, Azure Search, Redis, Elasticsearch, PostgreSQL) and by db.vector.id I mean the identifier of the vector (most used in native vector db like Pinecone, Qdrant, Chroma, Milvus).
Basically these attributes, name and ID, are used to identify the vector, maybe a better naming for
db.vector.namecan bedb.vector.field_name?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the context!
I agree that
db.vector.field_namewould be more descriptive.It could be difficult to collect it though - for cosmos, postgres and many other dbs it would require parsing queries (and specific parsing for vector search). I.e. it'd probably mean that very few generic DB instrumentations will do it.
Even the native instrumentation that we have in CosmosDB does not know if query supplied by user is doing vector search.
So I wonder how critical this attribute is for the observability purposes. Also I'd be mostly static - is it important enough to justify additional costs of populating it on each span?
A typical semconv decision making hint is: if we're not sure we need it, let's not add it. Adding attributes to spans is easy, removing them is breaking and hard.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still don't understand what's behind
db.vector.id- it seems to be a generic record id and there is nothing vector-specific here.I support adding generic attribute like
db.record.id: stringordb.record.ids: string[](needs polishing)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
db.vector.idis the record id but specific for the vector. Thedb.record.idwill work as well, since it's just an identifier for the record (i.e. vector).I think we should add
db.vector.field_namesince it is used in many database to specify which field has been used for the embedding.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is it the number of dimensions? let's call it something like
db.vector.dimension_countand also update the briefThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ezimuel you marked this as resolved, but looks like you missed this change, it still shows
dimensioninstead ofdimention_countThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in a3330ff
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should be captured with
gen_ai.request.model(and other gen-ai attributes) - it's ok to mix different attributes on the same telemetry item.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the
modelfor the embedding and can be different from thegen_ai.request.model. Vector databases are not strictly related with GenAI. There are many use cases where vector db can be used without GenAI (e.g. semantic search).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you provide some examples of databases that can compute embeddings?
How would database instrumentation know which model was used to create embeddings if it only stores and queries them?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some vendors that provide in-database embedding:
I think many other vendors will add this feature. Having the possibility to generate embeddings in the database simplify the customer use cases.
Regarding the question on how database instrumentation know which model was used to create an embedding this is an information that is specified when you create a collection or when you provide some search (e.g. in PostgreML). Moreover, I think the instrumentation libraries can also leverage this
modelattribute since they know the embedding model used to generate the vectors.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems all of them use some GenAI model (in-process or external) and provide an integration layer with it.
I'm not sure what benefit defining a new attribute for databases brings. If embeddings become cross-domain concerns, let's find a generic attribute name for the vectorization model that will be reused between DB and GenAI. For now I strongly recommend to avoid adding new attribute.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I agree that we can wait to see how this topic progresses.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should come up with a more common attribute not specific to vector dbs.
Many databases allow to limit number of returned rows:
Suggesting
db.query.max_returned_items. The actual returned count could be even better -db.query.item_countcould mean items inserted or returned depending on the operation.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lmolkova I see the similarity here but I think the
db.vector.query.top-kis more specific from a semantic point of view and more related to vectors, since it specifies the top k results in order, starting from the most similar. In semantic search we have this similarity value that is always present in any result that we don't have in standard database. Thelimitparameter of SQL returns the first k results but not in order, it depends on how you build the query (e.g. usingORDER BY).I personally think we should keep
db.vector.query.top-kand potentially add adb.query.limit(ordb.query.max_returned_itemsas you suggested) in a separate PR.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Out of the few dbs I checked, they use
limitin vector searchlimit- same with cosmos and other sql databasesSo we're saying that DB instrumentations will need to detect if query is related to vector search or not and depending on this populate top-k or limit. That's difficult or impossible, but most importantly inconsistent and depends on instrumentation capabilities.
I.e. instrumentations that don't have vector-db specifics and those that do will use different attributes for the same thing.
So, I'd still prefer
db.query.limitor something similar (and it should be under the same condition asdb.query.text- we cannot require instrumentations to do query parsing)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lmolkova do you agree that
top-kandlimitare two different concepts, based on my previous comment? If they are I think we cannot use a single attribute (e.g.db.query.limit) to manage both.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lmolkova just a reminder for this, thanks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ezimuel I see that databases use both terms to describe the same thing (see my comment above).
Let's say you have a postgres query like
SELECT * FROM items ORDER BY embedding <-> '[3,1,2]' LIMIT 5;- the general-purpose DB instrumentation can reportlimit. If you make it understand vector search syntax, it may be able to usetop_kinstead, but that's would be inconsistent and unfamiliar for those who use vector search in postgres.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lmolkova I see your point but I think
top-khas a different meaning fromlimit. If you are using a relation database as vector db,limitis fine since you are building an SQL statement and you specify the order. But, if you are using a native vector database (e.g. Qdrant), thetop-kis more relevant sincetopimplies the order, using a similarity metric.I think we should add both:
db.query.limitdb.vector.query.top-kCheck failure on line 595 in model/registry/db.yaml