Skip to content

Commit

Permalink
[Vectorize] Complete Vectorize 2 limits (#16497)
Browse files Browse the repository at this point in the history
* [Vectorize] Complete Vectorize 2 limits

* [Vectorize] Mention metadata limits in intro and Metadata docs
  • Loading branch information
netgusto authored Sep 18, 2024
1 parent 34e731c commit dea8093
Show file tree
Hide file tree
Showing 3 changed files with 22 additions and 2 deletions.
10 changes: 9 additions & 1 deletion src/content/docs/vectorize/get-started/intro.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ Specifically:

## 4. [Optional] Create metadata indexes

Vectorize allows you to add metadata along with vectors into your index, and also provides the ability to to filter on the vector metadata while querying vectors. To do so you would need to specify a metadata field as a "metadata index" for your Vectorize index.
Vectorize allows you to add up to 10KiB of metadata per vector into your index, and also provides the ability to filter on that metadata while querying vectors. To do so you would need to specify a metadata field as a "metadata index" for your Vectorize index.

:::note[When to create metadata indexes?]

Expand Down Expand Up @@ -168,6 +168,14 @@ npx wrangler vectorize list-metadata-index tutorial-index
└──────────────┴────────┘
```

You can create up to 10 metadata indexes per Vectorize index.

For metadata indexes of type `number`, the indexed number precision is that of float64.

For metadata indexes of type `string`, each vector indexes the first 64B of the string data truncated on UTF-8 character boundaries to the longest well-formed UTF-8 substring within that limit, so vectors are filterable on the first 64B of their value for each indexed property.

See [Vectorize Limits](/vectorize/platform/limits/) for a complete list of limits.

## 5. Insert vectors

Before you can query a vector database, you need to insert vectors for it to query against. These vectors would be generated from data (such as text or images) you pass to a machine learning model. However, this tutorial will define static vectors to illustrate how vector search works on its own.
Expand Down
2 changes: 2 additions & 0 deletions src/content/docs/vectorize/platform/limits.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ The following limits apply to accounts, indexes and vectors (as specified):
| Maximum namespaces per index | 1000 namespaces <sup>beta</sup> |
| Maximum namespace name length | 64 bytes |
| Maximum vectors upload size | 100 MB |
| Maximum metadata indexes per Vectorize index | 10 |
| Maximum indexed data per metadata index per vector | 64 bytes |

<sup>beta</sup> This limit is beta only and is expected to increase over time.

Expand Down
12 changes: 11 additions & 1 deletion src/content/docs/vectorize/reference/metadata-filtering.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import { Render, PackageManagers } from "~/components";

:::note[Enable metadata filtering]

Vectorize requires metadata indexes to be specified before vectors are inserted to support metadata filtering. Please refer to [Create metadata indexes](/vectorize/get-started/intro/#4-optional-create-metadata-indexes) for details.
Vectorize requires metadata indexes to be specified before vectors are inserted to support metadata filtering. `string`, `number` and `boolean` metadata indexes are supported. Please refer to [Create metadata indexes](/vectorize/get-started/intro/#4-optional-create-metadata-indexes) for details.

Vectorize supports [namespace](/vectorize/best-practices/insert-vectors/#namespaces) filtering by default.

Expand All @@ -19,6 +19,16 @@ In addition to providing an input vector to your query, you can also filter by [

By using metadata filtering to limit the scope of a query, you can filter by specific customer IDs, tenant, product category or any other metadata you associate with your vectors.

## Limits

You can store up to 10KiB of metadata per vector, and create up to 10 metadata indexes per Vectorize index.

For metadata indexes of type `number`, the indexed number precision is that of float64.

For metadata indexes of type `string`, each vector indexes the first 64B of the string data truncated on UTF-8 character boundaries to the longest well-formed UTF-8 substring within that limit, so vectors are filterable on the first 64B of their value for each indexed property.

See [Vectorize Limits](/vectorize/platform/limits/) for a complete list of limits.

## Supported operations

Optional `filter` property on `query()` method specifies metadata filter:
Expand Down

0 comments on commit dea8093

Please sign in to comment.