Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PIXELS-580] implement using LSH index to perform approximate nearest neighbour search on vector column #88

Open
wants to merge 23 commits into
base: master
Choose a base branch
from
Open

[PIXELS-580] implement using LSH index to perform approximate nearest neighbour search on vector column #88

wants to merge 23 commits into from

Commits on Jan 16, 2024

  1. implement a simple trino udf to calcualte the sum of all the elements… 

    … an input vector
    @TiannanSha
    TiannanSha committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    f88dedb View commit details
    Browse the repository at this point in the history

  2. use array instead of features because apparently trino features don't… 

    … support more than ten features
    @TiannanSha
    TiannanSha committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    5a6d75c View commit details
    Browse the repository at this point in the history

  3. implement three types of distances: euclidean, dotproduct and cosine … 

    …similarity
    @TiannanSha
    TiannanSha committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    df04bbf View commit details
    Browse the repository at this point in the history

  4. make select exactNNS() udf work in trino

    @TiannanSha
    TiannanSha committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    2300020 View commit details
    Browse the repository at this point in the history

  5. fix a test

    @TiannanSha
    TiannanSha committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    1767b44 View commit details
    Browse the repository at this point in the history

  6. fix a comment

    @TiannanSha
    TiannanSha committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    d52a31e View commit details
    Browse the repository at this point in the history

  7. minor polish

    @TiannanSha
    TiannanSha committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    2201829 View commit details
    Browse the repository at this point in the history

  8. make pixels vector type support trino array type

    @TiannanSha
    TiannanSha committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    420408a View commit details
    Browse the repository at this point in the history

  9. implement a exact NNS that acts as an aggregation function and should… 

    … work on a multinode environment
    @TiannanSha
    TiannanSha committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    abbbee7 View commit details
    Browse the repository at this point in the history

  10. clean up

    @TiannanSha
    TiannanSha committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    4ca554d View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2024

  1. pretty much finished LSH build; LSH search wip

    @TiannanSha
    TiannanSha committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    5bc07ba View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2024

  1. wip: lsh search

    @TiannanSha
    TiannanSha committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    01a3903 View commit details
    Browse the repository at this point in the history

Commits on Feb 5, 2024

  1. lshSearch work in progress

    @TiannanSha
    TiannanSha committed Feb 5, 2024
    Configuration menu
    Copy the full SHA
    19fedc8 View commit details
    Browse the repository at this point in the history

Commits on Feb 9, 2024

  1. implement LSH search, including updating mapping from col to buckets … 

    …and persist the mapping when JVM shuts down
    @TiannanSha
    TiannanSha committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    3e89598 View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2024

  1. fix the ser and deser for LSH index

    @TiannanSha
    TiannanSha committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    0fc8cd1 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2024

  1. auto decide s3dir for storing LSH buckets. clean up

    @TiannanSha
    TiannanSha committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    9150fab View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. implement code for experiments; before fixing remote page too large

    @TiannanSha
    TiannanSha committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    dd9016d View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. accidentally got ignored entire folder of lsh build

    @TiannanSha
    TiannanSha committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    b843296 View commit details
    Browse the repository at this point in the history

  2. fix lsh_build

    @TiannanSha
    TiannanSha committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    fad3f5d View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2024

  1. implement LSH load and adjust lsh search

    @TiannanSha
    TiannanSha committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    9eaf254 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. fix lsh_search() so that it works with lsh_load()

    @TiannanSha
    TiannanSha committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    09afb3f View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2024

  1. minor changes on lsh_load()

    @TiannanSha
    TiannanSha committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    f231da9 View commit details
    Browse the repository at this point in the history

  2. add bucket write threshold to lsh_build() to avoid sending big messages

    @TiannanSha
    TiannanSha committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    bbbb793 View commit details
    Browse the repository at this point in the history