Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: support refreshing Iceberg tables #5707

Open
wants to merge 40 commits into
base: main
Choose a base branch
from
Open

feat: support refreshing Iceberg tables #5707

wants to merge 40 commits into from

Commits on Jul 2, 2024

  1. Initial commit of refreshing Iceberg.

    @lbooker42
    lbooker42 committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    470b09c View commit details
    Browse the repository at this point in the history

  2. Rebased to main.

    @lbooker42
    lbooker42 committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    a8d957a View commit details
    Browse the repository at this point in the history

  3. Change IcebergInstructions refreshing indicator to enum instead of bo… 

    …olean
    @lbooker42
    lbooker42 committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    264fdb1 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. WIP, for review

    @lbooker42
    lbooker42 committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    58d0a73 View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. Manual and auto-refreshing working, better documentation.

    @lbooker42
    lbooker42 committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    e090474 View commit details
    Browse the repository at this point in the history

  2. Addressed more PR comments, some remaining.

    @lbooker42
    lbooker42 committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    57021ad View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. WIP, some PR comments addressed.

    @lbooker42
    lbooker42 committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    fb882e8 View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2024

  1. WIP, even more PR comments addressed.

    @lbooker42
    lbooker42 committed Jul 27, 2024
    Configuration menu
    Copy the full SHA
    5bbdeb2 View commit details
    Browse the repository at this point in the history

  2. Nearly all PR comments addressed.

    @lbooker42
    lbooker42 committed Jul 27, 2024
    Configuration menu
    Copy the full SHA
    3da205c View commit details
    Browse the repository at this point in the history

  3. merged with main

    @lbooker42
    lbooker42 committed Jul 27, 2024
    Configuration menu
    Copy the full SHA
    91acf9b View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. Adjustment to IcebergInstructions update mode.

    @lbooker42
    lbooker42 committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    08dd329 View commit details
    Browse the repository at this point in the history

  2. Added python wrapper for Iceberg refreshing tables.

    @lbooker42
    lbooker42 committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    7af0d1d View commit details
    Browse the repository at this point in the history

  3. Changes to mocked tests for ColumnSourceManager and PartitionAwareSou… 

    …rceTable.
    @lbooker42
    lbooker42 committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    2d79c38 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. Added DHError handler and add'l documentation to python snapshots()… 

    … call.
    @lbooker42
    lbooker42 committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    3809f21 View commit details
    Browse the repository at this point in the history

  2. Fixed typo in JavaDoc

    @lbooker42
    lbooker42 committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    5273a15 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. WIP

    @lbooker42
    lbooker42 committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    b9e2c6e View commit details
    Browse the repository at this point in the history

  2. Suggestion from review

    @lbooker42
    lbooker42 committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    9937f79 View commit details
    Browse the repository at this point in the history

  3. WIP, changes to revert some transaction token code.

    @lbooker42
    lbooker42 committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    cd08038 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2024

  1. Correct logic across multiple transactions.

    @lbooker42
    lbooker42 committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    f28325f View commit details
    Browse the repository at this point in the history

  2. Merged with main

    @lbooker42
    lbooker42 committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    2d92b3f View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. Moved transaction accumulation to AbstractTableLocationProvider

    @lbooker42
    lbooker42 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    cd31d82 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. Moved transaction accumulation to AbstractTableLocationProvider

    @lbooker42
    lbooker42 committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    d680c0c View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2024

  1. PR comments addressed.

    @lbooker42
    lbooker42 committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    6607fc3 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. Updated to use IcebergTableAdapter and exposed in python. Addressed P… 

    …R comments.
    @lbooker42
    lbooker42 committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    893336f View commit details
    Browse the repository at this point in the history

  2. Incorporated external PR to update PartitioningColumnDataIndex for re… 

    …freshing tables.
    @lbooker42
    lbooker42 committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    68e4546 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. Added additional snapshots with removes to IcebergToolsTest resources.

    @lbooker42
    lbooker42 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    273f5c1 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into lab-iceberg-refreshing

    @lbooker42
    lbooker42 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    1e92a19 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. Manual and auto refreshing tests for Iceberg.

    @lbooker42
    lbooker42 committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    f72c1b7 View commit details
    Browse the repository at this point in the history

  2. Manual and auto refreshing tests for Iceberg, not passing.

    @lbooker42
    lbooker42 committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    5c7ff12 View commit details
    Browse the repository at this point in the history

  3. PR comments addressed.

    @lbooker42
    lbooker42 committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    92eec61 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. Implemented improved location reference counting in AbstractTableLoca… 

    …tionProvider and RegionedColumnSourceManager
    @lbooker42
    lbooker42 committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    95194b7 View commit details
    Browse the repository at this point in the history

  2. Fixing doc problem.

    @lbooker42
    lbooker42 committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    09e2b6e View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. For review only, does not compile :(

    @lbooker42
    lbooker42 committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    30910dd View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. Compiles now, still many problems

    @lbooker42
    lbooker42 committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    dd12240 View commit details
    Browse the repository at this point in the history

  2. Working through problems.

    @lbooker42
    lbooker42 committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    944dac6 View commit details
    Browse the repository at this point in the history

  3. Cleanup and minor changes

    @lbooker42
    lbooker42 committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    912b9f2 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Refactored ATLP

    @lbooker42
    lbooker42 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    e2b6fd0 View commit details
    Browse the repository at this point in the history

  2. Merged with main.

    @lbooker42
    lbooker42 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    72b03be View commit details
    Browse the repository at this point in the history

  3. Updated but RCSM still not referenced properly.

    @lbooker42
    lbooker42 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    01e50fe View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. Refreshing tests still need work.

    @lbooker42
    lbooker42 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    81e88d8 View commit details
    Browse the repository at this point in the history