Skip to content

HTTPFS: Move from HTTPException to base class IOException#9351

Merged
Mytherin merged 3 commits intoduckdb:mainfrom
carlopi:httpfs_step_1
Oct 16, 2023
Merged

HTTPFS: Move from HTTPException to base class IOException#9351
Mytherin merged 3 commits intoduckdb:mainfrom
carlopi:httpfs_step_1

Conversation

@carlopi
Copy link
Contributor

@carlopi carlopi commented Oct 15, 2023

This commit might need to be back-ported (at least for MacOS builds) to previous released httpfs extension binaries (most affected are v0.9.1 due to how this is put onto the spotlight due to #9286) to fix issue like #9340

My idea on how to proceed is as follow:

  1. this can be merged into duckdb and have nightly binaries (and connected extensions) throwing IOExceptions (that are properly handled) instead of specialised payload of HTTPExtensions
    This allows to test (on CI and on nightlies) whether this solves the actual problem
    This also allows, independently, to explore how to build specific extensions with a custom patch (this commit is basically
    the patch)
  2. Fixing the handling of HTTPExtension in core duckdb. Potentially this could mean axing the HTTP-specific payload and moving to a generic textual plus custom payload extracted via virtual function calls

This commit might need to be back-ported (at least for MacOS builds) to previous released httpfs extension binaries (most affected are v0.9.1 due to how this is put onto the spotlight due to duckdb#9286) to fix issue like duckdb#9340

My idea on how to proceed is as follow:

this can be merged into duckdb and have nightly binaries (and connected extensions) throwing IOExceptions (that are properly handled) instead of specialised payload of HTTPExtensions
This allows to test (on CI and on nightlies) whether this solves the actual problem
This also allows, independently, to explore how to build specific extensions with a custom patch (this commit is basically
the patch)
Fixing the handling of HTTPExtension in core duckdb. Potentially this could mean axing the HTTP-specific payload and moving to a generic textual plus custom payload extracted via virtual function calls
@github-actions github-actions bot marked this pull request as draft October 15, 2023 15:20
@carlopi carlopi marked this pull request as ready for review October 15, 2023 15:20
@github-actions github-actions bot marked this pull request as draft October 15, 2023 17:10
@carlopi carlopi marked this pull request as ready for review October 15, 2023 17:10
@github-actions github-actions bot marked this pull request as draft October 15, 2023 19:10
@carlopi carlopi marked this pull request as ready for review October 15, 2023 19:13
@Mytherin Mytherin merged commit bdc5978 into duckdb:main Oct 16, 2023
@Mytherin
Copy link
Collaborator

Thanks!

@carlopi carlopi deleted the httpfs_step_1 branch October 16, 2023 07:16
samansmink added a commit to duckdb/duckdb_httpfs_wasm_experiment that referenced this pull request Oct 17, 2023
Mytherin added a commit to Mytherin/duckdb that referenced this pull request Jan 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants