Fixed BUG: Handling insertion with zero vectors

[swetavooda]

What is the Issue?

Pinecone insists that dense vectors cannot be zero in all dimensions. When building/inserting a zero vector pinecone will throw an error.

At Build Time (indexing)

Current Working
During build (when there are zero vectors in the table): ERROR is thrown and values are not pushed into pinecone.

Possible ways to handle:

1. Remove the zero vectors and push the remaining vectors into pinecone(remove zero vectors) - but this can be a bad user experience as we are deleting vectors without the permission of the user.
2. Chosen: Do not let the user index(build) until there are no zero vectors.

During Insertion (Buffering)

Current Working
The user can insert zero vectors into the indexed table, but only when the buffer is flushed the error occurs and the the remaining vectors are also paused from flushing into pinecone.

Problem with this:

• If the user sees and error message he would need to remove the zero vector manually, which can be stopped at the insertion stage itself.
• The zero vector is added to the Buffer and the value of the id doesn't change. on addition of every other vector the same error will be thrown(bad user experience) and the buffer wont be flushed since the check point bklnumber doesn't change.

Solution

1. Remove the tuple from buffer and update the bklnumber counter for flushing.
2. Better solution: Do not allow users insert zero vector (thereby not adding the tuple to the buffer to avoid complexities).

[Chitti-Ankith]

Can you add a test-case for this as well? The logic LGTM

[swetavooda]

Added mock test case for zero vector insertions:

• Build after insertions
• Insert after build

Additional comments on the logic

Validating every vector on insert + flushing + build would be redundant and slows down the insertion
[TODO]Alternative approach:

• Validate only on flushing + build time
• Skip zero vector: not to block user from the remaining insertion and throw a WARN instead of a blocking ERROR