Use curl as optional client v1.4 by Tmonster · Pull Request #96 · duckdb/duckdb-httpfs

Tmonster · 2025-08-13T08:28:23Z

After duckdb/duckdb#18107 landed in duckdb/duckdb, and moving the duckdb submodule to a recent commit on v1.4-andium, this PR allows to switch at runtime based on the newly added httpfs config option httpfs_client_implementation:

D SET logging_storage=stdout;
D PRAGMA enable_logging('HTTP');
D SET httpfs_client_implementation='default';
D select count(*)
  FROM 'https://github.com/duckdb/duckdb-data/releases/download/v1.0/event_baserunning_advance_attempt.parquet';
[LOG] 2025-07-03 10:06:18.479, HTTP, DEBUG, {'request': {'type': HEAD, 'url': 'https://github.com/duckdb/duckdb-data/releases/download/v1.0/event_baserunning_advance_attempt.parquet', 'headers': {}}, 'response': {'status': OK_200, 'reason': OK, 'headers': {X-Timer='S1751537178.169255,VS0,VE1', x-ms-request-id=91401b99-a01e-000c-7caa-dd765d000000, X-Served-By='cache-iad-kcgs7200132-IAD, cache-rtm-ehrd2290029-RTM', Fastly-Restarts=1, x-ms-lease-status=unlocked, x-ms-creation-time='Tue, 17 Jan 2023 21:28:40 GMT', x-ms-blob-type=BlockBlob, x-ms-blob-content-md5='0OgVgULsCWfa1mU4BlFEbg==', X-Cache-Hits='3730, 1', Via='1.1 varnish, 1.1 varnish', Server=Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0, x-ms-version=2025-05-05, X-Cache='HIT, HIT', ETag='"0x8DAF8D1CD43CA79"', Connection=keep-alive, x-ms-lease-state=available, Last-Modified='Tue, 17 Jan 2023 21:28:40 GMT', Date='Thu, 03 Jul 2025 10:06:18 GMT', Content-Length=21916382, Content-Type=application/octet-stream, Content-Disposition='attachment; filename=event_baserunning_advance_attempt.parquet', x-ms-server-encrypted=true, Age=806, Accept-Ranges=bytes}}}, CONNECTION, 2, 11, NULL
┌────────────────┐
│  count_star()  │
│     int64      │
├────────────────┤
│    9388600     │
│ (9.39 million) │
└────────────────┘
D SET httpfs_client_implementation='curl';
D select count(*)
  FROM 'https://github.com/duckdb/duckdb-data/releases/download/v1.0/event_baserunning_advance_attempt.parquet';
[LOG] 2025-07-03 10:06:30.247, HTTP, DEBUG, {'request': {'type': HEAD, 'url': 'https://github.com/duckdb/duckdb-data/releases/download/v1.0/event_baserunning_advance_attempt.parquet', 'headers': {}}, 'response': {'status': OK_200, 'reason': '', 'headers': {content-type=application/octet-stream, x-ms-lease-state=available, last-modified='Tue, 17 Jan 2023 21:28:40 GMT', x-ms-request-id=91401b99-a01e-000c-7caa-dd765d000000, accept-ranges=bytes, x-ms-version=2025-05-05, server=Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0, x-ms-creation-time='Tue, 17 Jan 2023 21:28:40 GMT', x-ms-blob-content-md5='0OgVgULsCWfa1mU4BlFEbg==', x-cache='HIT, HIT', __RESPONSE_STATUS__='HTTP/2 200 ', etag='"0x8DAF8D1CD43CA79"', x-ms-blob-type=BlockBlob, x-ms-server-encrypted=true, age=818, x-ms-lease-status=unlocked, x-served-by='cache-iad-kcgs7200132-IAD, cache-rtm-ehrd2290021-RTM', fastly-restarts=1, via='1.1 varnish, 1.1 varnish', date='Thu, 03 Jul 2025 10:06:30 GMT', x-cache-hits='3730, 1', content-disposition='attachment; filename=event_baserunning_advance_attempt.parquet', x-timer='S1751537190.940711,VS0,VE1', content-length=21916382}}}, CONNECTION, 2, 13, NULL
┌────────────────┐
│  count_star()  │
│     int64      │
├────────────────┤
│    9388600     │
│ (9.39 million) │
└────────────────┘
D SET httpfs_client_implementation='httplib';
D select count(*)
  FROM 'https://github.com/duckdb/duckdb-data/releases/download/v1.0/event_baserunning_advance_attempt.parquet';
[LOG] 2025-07-03 10:07:45.552, HTTP, DEBUG, {'request': {'type': HEAD, 'url': 'https://github.com/duckdb/duckdb-data/releases/download/v1.0/event_baserunning_advance_attempt.parquet', 'headers': {}}, 'response': {'status': OK_200, 'reason': OK, 'headers': {X-Timer='S1751537265.144944,VS0,VE0', x-ms-request-id=91401b99-a01e-000c-7caa-dd765d000000, X-Served-By='cache-iad-kcgs7200132-IAD, cache-rtm-ehrd2290047-RTM', Fastly-Restarts=1, x-ms-lease-status=unlocked, x-ms-creation-time='Tue, 17 Jan 2023 21:28:40 GMT', x-ms-blob-type=BlockBlob, x-ms-blob-content-md5='0OgVgULsCWfa1mU4BlFEbg==', X-Cache-Hits='3730, 0', Via='1.1 varnish, 1.1 varnish', Server=Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0, x-ms-version=2025-05-05, X-Cache='HIT, HIT', ETag='"0x8DAF8D1CD43CA79"', Connection=keep-alive, x-ms-lease-state=available, Last-Modified='Tue, 17 Jan 2023 21:28:40 GMT', Date='Thu, 03 Jul 2025 10:07:45 GMT', Content-Length=21916382, Content-Type=application/octet-stream, Content-Disposition='attachment; filename=event_baserunning_advance_attempt.parquet', x-ms-server-encrypted=true, Age=893, Accept-Ranges=bytes}}}, CONNECTION, 2, 15, NULL
┌────────────────┐
│  count_star()  │
│     int64      │
├────────────────┤
│    9388600     │
│ (9.39 million) │
└────────────────┘
D SET httpfs_client_implementation='something_else';
Invalid Input Error:
Unsupported option for httpfs_client_implementation, only `curl`, `httplib` and `default` are currently supported
It can be checked from the headers that slightly different implementations are used, given for example different styling for Etag vs etag or similar implementation details.

…ader

…t overwritten on the stack

Yes this will fix my build errors at duckdb/duckdb-httpfs#96. This CI link has a passing build status https://github.com/duckdb/duckdb-httpfs/actions/runs/16991311944/job/48171130825 ~~[DO NOT MERGE]: I am testing to see if this is the correct fix with [this PR](duckdb/duckdb-httpfs#96) first. I am just updating the duckdb submodule pointer for the httpfs fork to the branch here. If those tests pass then I know what the correct fix is. (don't know how to trigger it otherwise yet)~~ This is prompted by this PR duckdb/duckdb-httpfs#96. Related [CI failure](https://github.com/duckdb/duckdb-httpfs/actions/runs/16932639981/job/47981684022?pr=96#step:26:597) Seems like the httplib has conflicts with the max() function. I've searched for other instances of `::max()` and `::min()` in httplib.hpp and didn't find any. It seems like the proper fix is to use `(std::numeric_limits<size_t>::max)()` as seen on line [96 of httplib.hpp](https://github.com/duckdb/duckdb/blob/1f0de28806a8915c8203dd060dad549f28f5539b/third_party/httplib/httplib.hpp#L96) and that did not fail the windows build

carlopi · 2025-08-19T10:28:06Z

I remember there was a problem when running:

SET httpfs_client_implementation='curl';
INSTALL non_existent_extension;

throwing an uncaught error, that likely signal a problem elsewhere.

Could you check whether that's still the case?

Apart from that, also considering this is not the default AND it's on 1.4- branch, I would think this is good to be merged.

carlopi

Thanks, looks great to have this!

carlopi · 2025-08-19T10:33:58Z

Other things to consider, as follow ups:

running all duckdb test suite with a configuration to force curl as default (when I did that last time only error with INSTALL extensions)
testing on a physical Windows machine
considering whether the default should be curl (OSX + Linux?, likely 1.5)
documentation

carlopi · 2025-08-21T09:36:32Z

Should we merge or is there any blocker?

I think merge + bumping duckdb-httpfs hash in duckdb/duckdb would make this more properly available, other steps such as docs can be done then before 1.4.0

Bump Httpfs so that duckdb/duckdb-httpfs#96 is included. Allows us to add more testing for httpfs using curl.

naah69 · 2025-09-03T05:54:06Z

link #82

dentiny · 2025-09-27T00:07:35Z

extension/httpfs/httpfs_curl_client.cpp

+
+		CURLcode res;
+		{
+			curl_easy_setopt(*curl, CURLOPT_URL, request_info->url.c_str());


curious should we sanity check the return value for option setting?
http://curl.se/libcurl/c/curl_easy_setopt.html

dentiny · 2025-09-27T00:08:33Z

extension/httpfs/httpfs_curl_client.cpp

+};
+
+
+static idx_t httpfs_client_count = 0;


curious why don't we use std::atomic<idx_t> here? So we don't need another global variable (in two places...) and prob slightly better perf

Tmonster added 30 commits May 16, 2025 12:59

include open ssl

29052b3

attempt to overwrite get

180bcb6

compiles with curl now

303c9a7

put head delete post get replaced

b8272b1

bump duckdb

e6c2780

remove extension stuff

2afdc88

install curl during build

be81a80

run on ubuntu-latest

fdac277

init vcpackage toolchain path

ef581de

actually add params to end of url for get requests

e7bc7e0

get the response error message as well from the http response code he…

b74abc2

…ader

remove all running httplib.hpp code

88f96fc

add back in state initialization

04cb2a8

remove output result

02e8ac1

add json read test

c92998c

add core extensions var

fa93e10

always add headers, and set many more curl options on initialize

499de90

update extension ci tools

9d8f71e

fix test add comments

e0cce43

declare header object on heap not on stack

9b90587

also uniq string

5b3d988

some more debugging statements

ccbf00a

move a lot of the request response info to the heap so it does not ge…

de6d606

…t overwritten on the stack

this should work on the build now

0c167fd

more clean up

5d8d79f

Merge branch 'main' into use_curl_as_client

d901fc4

call curl_global_init, unsure when to call cleanup

0c42015

remove verbose

bd44ff2

reuse many components

3da7bff

add test to check response headers

198288e

Tmonster added 9 commits August 15, 2025 15:12

update submodule again

43fcbcd

some changes to the workflow

5ce3275

update duckdb ref

87f9797

update ref

6a44033

fix the workflow

8608552

do not set duckdb to ref of calling worfklow

07dbd11

needs to be my duckdb repo

f77d92c

fix up main distribution pipeline

ab630dc

use duckdb v1.4

661c861

bump duckdb to include build fix

4e5a985

carlopi approved these changes Aug 19, 2025

View reviewed changes

Tmonster added 2 commits August 20, 2025 11:28

if there is no content handler, do not call it

13972ba

change expected error message to pass tests

0bb4efa

Tmonster mentioned this pull request Aug 20, 2025

Use curl as optional client #77

Closed

samansmink approved these changes Aug 21, 2025

View reviewed changes

samansmink merged commit c31f9e9 into duckdb:v1.4-andium Aug 21, 2025
23 checks passed

Tmonster mentioned this pull request Aug 22, 2025

bump httpfs so it includes curl option duckdb/duckdb#18691

Merged

carlopi mentioned this pull request Aug 24, 2025

Replace HTTPlib client with Curl #58

Closed

hannes added a commit to duckdb/duckdb that referenced this pull request Aug 31, 2025

bump httpfs so it includes curl option (#18691)

c545c08

Bump Httpfs so that duckdb/duckdb-httpfs#96 is included. Allows us to add more testing for httpfs using curl.

naah69 mentioned this pull request Sep 3, 2025

Is httpfs support http connection pool？ #82

Open

carlopi mentioned this pull request Sep 4, 2025

Use curl as optional client #86

Closed

dentiny reviewed Sep 27, 2025

View reviewed changes

Y-- mentioned this pull request Jan 30, 2026

Server certificate verification fails in some environments (was: Error 500, can't access the UI) duckdb/duckdb-ui#95

Open

guillesd mentioned this pull request Feb 16, 2026

Request to test network stack changes in duckdb-httpfs between v1.4 and v1.5 duckdb/duckdb#20977

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use curl as optional client v1.4#96

Use curl as optional client v1.4#96
samansmink merged 66 commits intoduckdb:v1.4-andiumfrom
Tmonster:use_curl_as_optional_client_v1.4

Tmonster commented Aug 13, 2025 •

edited

Loading

Uh oh!

carlopi commented Aug 19, 2025 •

edited

Loading

Uh oh!

carlopi left a comment

Uh oh!

carlopi commented Aug 19, 2025

Uh oh!

carlopi commented Aug 21, 2025

Uh oh!

Uh oh!

naah69 commented Sep 3, 2025

Uh oh!

dentiny Sep 27, 2025

Uh oh!

dentiny Sep 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Tmonster commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlopi commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlopi left a comment

Choose a reason for hiding this comment

Uh oh!

carlopi commented Aug 19, 2025

Uh oh!

carlopi commented Aug 21, 2025

Uh oh!

Uh oh!

naah69 commented Sep 3, 2025

Uh oh!

dentiny Sep 27, 2025

Choose a reason for hiding this comment

Uh oh!

dentiny Sep 27, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Tmonster commented Aug 13, 2025 •

edited

Loading

carlopi commented Aug 19, 2025 •

edited

Loading