Support similarity scores in Document API #1794
Conversation
| * @author Thomas Vitale | ||
| * @since 1.0.0 | ||
| */ | ||
| public enum DocumentMetadata { |
There was a problem hiding this comment.
The idea for this enum is to use it for other common metadata used in Documents, such as the "source file" or "page" when using a DocumentReader, helping the RAG flow traceability.
| * The lower the distance, the more they are similar. | ||
| * It's the opposite of the similarity score. | ||
| */ | ||
| DISTANCE("distance"); |
There was a problem hiding this comment.
I kept this metadata for backward compatibility, but we might consider removing it completely since we now have the "score" field in each Document (and "distance" is always the opposite value of "score").
| .filter(s -> s.score >= request.getSimilarityThreshold()) | ||
| .sorted(Comparator.<Similarity>comparingDouble(s -> s.score).reversed()) | ||
| .peek(document -> document | ||
| .setScore(EmbeddingMath.cosineSimilarity(userQueryEmbedding, document.getEmbedding()))) |
There was a problem hiding this comment.
If we remove the "embedding" field (see: #1781), the SimpleVectorStore will not work
| // It must always be "latest" or else Azure locks the image after a while. See: | ||
| // https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/60 | ||
| public static final DockerImageName DEFAULT_IMAGE = DockerImageName | ||
| .parse("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest"); |
There was a problem hiding this comment.
I hope we'll be able to have integration tests for CosmosDB based on Testcontainers in the future. For now, this image includes the vector store-specific features disabled and there's no way to enable them, so it cannot be used.
| @@ -298,6 +303,8 @@ public static final class PineconeVectorStoreConfig { | |||
|
|
|||
| private final String contentFieldName; | |||
|
|
|||
| // TODO: Why is this field configurable? Can we remove this after standardizing | |||
There was a problem hiding this comment.
Would it be ok to remove this and keep the standard "distance" metadata? Having this configurable means we cannot use the metadata reliably across implementations.
| Map<String, Object> metadata = this.config.metadataFields.stream() | ||
| .map(MetadataField::name) | ||
| .filter(doc::hasProperty) | ||
| .collect(Collectors.toMap(Function.identity(), doc::getString)); | ||
| // TODO: this seems wrong. The key is named "vector_store", but the value is the |
There was a problem hiding this comment.
Would it be ok to remove this and keep the standard "distance" metadata?
|
PR updated after #1822 was merged |
Document * Introduced “score” attribute in Document API. It stores the similarity score. * Consolidate “distance” metadata for Documents. It stores the distance measurement. * Adopted prefix-less naming convention in Document.Builder and deprecated old methods. * Deprecated the many overloaded Document constructors in favour of Document.Builder. Vector Stores * Every vector store implementation now configures a “score” attribute with the similarity score of the Document embedding. It also includes the “distance” metadata with the distance measurement. * Fixed error in Elasticsearch where distance and similarity were mixed up. * Added missing integration tests for SimpleVectorStore. * The Azure Vector Store and HanaDB Vector Store do not include those measurements because the product documentation do not include information about how the similarity score is returned, and without access to the cloud products I could not verify that via debugging. * Improved tests to actually assert the result of the similarity search based on the returned score. Signed-off-by: Thomas Vitale <ThomasVitale@users.noreply.github.com>
ThomasVitale
force-pushed
the
similarity-score-documents
branch
from
467dbc3
to
efca3c7
Compare
Document
Vector Stores