Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: delta-io/delta-rs
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: python-v0.16.3
Choose a base ref
...
head repository: delta-io/delta-rs
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: python-v0.16.4
Choose a head ref
  • 11 commits
  • 42 files changed
  • 5 contributors

Commits on Mar 26, 2024

  1. fix(python): wrong batch size (#2314)

    # Description
    Was passing the wrong param
    
    - closes #2309
    ion-elgreco authored Mar 26, 2024
    Configuration menu
    Copy the full SHA
    5b404e2 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2024

  1. fix(rust): throw schema mismatch when decimal doesn't fit in the deci…

    …mal type of table (#2330)
    
    # Description
    The `can_cast_types` is a bit to flexible with decimals, since we define
    the decimal type in the table metadata. So if the source data contains a
    decimal that is larger than the decimal in the table metadata, we should
    throw an error. Before we would let our writer just write because
    `can_cast_types `does not check precision and scale for casting so
    Decimal(30.1) could be written after the table was set to only allow
    precision=1, scale=1.
    
    After the write, the json parser would not be able to read the table
    anymore since it throws a decimal overflow, and rightfully so. This fix
    will prevent tables to get in these invalid states.
    ion-elgreco authored Mar 28, 2024
    Configuration menu
    Copy the full SHA
    bb46c8a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e2aa8ba View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    01f832c View commit details
    Browse the repository at this point in the history
  4. fix: remove tmp files in cleanup_metadata (#2356)

    # Description
    We didn't clean up failed commits when possible.
    
    # Related Issue(s)
    - fixes #2351
    ion-elgreco authored Mar 28, 2024
    Configuration menu
    Copy the full SHA
    88937d3 View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2024

  1. fix(python,rust): optimize compact on schema evolved table (#2358)

    # Description
    Default should be always to add missing columns in compaction.
    
    # Related Issue(s)
    - fixes #2347
    ion-elgreco authored Mar 29, 2024
    Configuration menu
    Copy the full SHA
    fa435e1 View commit details
    Browse the repository at this point in the history
  2. feat(rust): derive Copy on some public enums (#2329)

    # Description
    This PR derives the `Copy` trait on public enums that had the `Clone`
    trait derived, when applicable.
    lasantosr authored Mar 29, 2024
    Configuration menu
    Copy the full SHA
    e58df28 View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2024

  1. fix: add config for parquet pushdown on delta scan (#2364)

    # Description
    Delta scan will push filter to the parquet scan when possible. Added a
    new configuration for the special case where operations need to operate
    on an entire file but still want to perform pruning.
    
    # Related Issue(s)
    - fixes #2362
    Blajda authored Mar 31, 2024
    Configuration menu
    Copy the full SHA
    7568b57 View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2024

  1. fix: add snappy compression on checkpoint files (#2365)

    # Description
    Noticed that we didn't set any compression, this should bring down the
    file size a bit for users that have large checkpoints
    ion-elgreco authored Apr 1, 2024
    Configuration menu
    Copy the full SHA
    e34767d View commit details
    Browse the repository at this point in the history
  2. chore(rust): bump datafusion to 36 (#2249)

    # Description
    The description of the main changes of your pull request
    
    # Related Issue(s)
    <!---
    For example:
    
    - closes #106
    --->
    
    # Documentation
    
    <!---
    Share links to useful documentation
    --->
    universalmind303 authored Apr 1, 2024
    Configuration menu
    Copy the full SHA
    2963692 View commit details
    Browse the repository at this point in the history
  3. chore: bump python 0.16.4 (#2371)

    # Description
    Another patch release
    ion-elgreco authored Apr 1, 2024
    Configuration menu
    Copy the full SHA
    dcd6be0 View commit details
    Browse the repository at this point in the history
Loading