Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(test): integration test for S3 log store with pyspark #1988

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Commits on Jan 30, 2024

  1. Configuration menu
    Copy the full SHA
    4eee451 View commit details
    Browse the repository at this point in the history
  2. Point pyspark test to the local dockjer-compose setup

    This still is not working, but it's not totally failing I guess
    rtyler committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    32e2c28 View commit details
    Browse the repository at this point in the history
  3. chore: upgrade to DataFusion 35.0 (delta-io#2121)

    # Description
    This PR upgrades `delta-rs` to using DataFusion 35.0, which was recently
    released. In order to do this, I had to fix a few breaking changes, and
    also upgrade Arrow to 50 and `sqlparser` to 0.41.
    
    # Related Issue(s)
    N/A
    
    # Documentation
    See here for the list of PRs which required code change:
    - apache/datafusion#8703
    -
    https://github.com/apache/arrow-datafusion/blob/ec6abece2dcfa68007b87c69eefa6b0d7333f628/dev/changelog/35.0.0.md?plain=1#L227
    
    ---------
    
    Co-authored-by: Ming Ying <ming.ying.nyc@gmail.com>
    2 people authored and rtyler committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    893d631 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2024

  1. fix(s3): restore working test for DynamoDb log store repair log on re…

    …ad (delta-io#2120)
    
    # Description
    
    Make sure the read path for delta table commit entries passes through
    the log store, enabling it to ensure the invariants and potentially
    repair a broken commit in the context of S3 / DynamoDb log store
    implementation.
    
    This also adds another test in the context of S3 log store: repairing a
    log store on load was not implemented previously.
      
    Note that this a stopgap and not a complete solution: it comes with a
    performance penalty as we're triggering a redundant object store list
    operation just for the purpose of "triggering" the log store
    functionality.
    
    
    fixes delta-io#2109
    
    ---------
    
    Co-authored-by: Ion Koutsouris <15728914+ion-elgreco@users.noreply.github.com>
    Co-authored-by: R. Tyler Croy <rtyler@brokenco.de>
    3 people committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    d1b24f5 View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2024

  1. Configuration menu
    Copy the full SHA
    abc2664 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2024

  1. Configuration menu
    Copy the full SHA
    1e1a12b View commit details
    Browse the repository at this point in the history