Broken pipe, when use Hadoop HDFS via WebHDFS REST API #8264
Description
I did this
- I have installed Hadoop HDFS (v. 3.1.3) with kerberos auth.
- I try to create file on HDFS via WebHDFS REST API with following command:
$ curl -sS -L -X PUT -T krb_spark-3.0.0.tar.gz --negotiate -u : "http://centos.hdfs.alt:50070/webhdfs/v1/user/test_dir_alt/krb_spark-3.0.0.tar.gz5?op=CREATE"
This command ends with an error:
curl: (55) Send failure: Broken pipe
Verbose output:
$ curl -v -sS -L -X PUT -T krb_spark-3.0.0.tar.gz --negotiate -u : "http://centos.hdfs.alt:50070/webhdfs/v1/user/test_dir_alt/krb_spark-3.0.0.tar.gz5?op=CREATE"
* Trying 10.88.17.66:50070...
* Connected to centos.hdfs.alt (10.88.17.66) port 50070 (#0)
* Server auth using Negotiate with user 'user'
> PUT /webhdfs/v1/user/test_dir_alt/krb_spark-3.0.0.tar.gz5?op=CREATE HTTP/1.1
> Host: centos.hdfs.alt:50070
> Authorization: Negotiate YIICqgYGKwYBBQUCoIICnjCCApqgDTALBgkqhkiG9xIBAgKiggKHBIICg2CCAn8GCSqGSIb3EgECAgEAboICbjCCAmqgAwIBBaEDAgEOogcDBQAgAAAAo4IBdWGCAXEwggFtoAMCAQWhChsISERGUy5BTFSiIjAgoAMCAQOhGTAXGwRIVFRQGw9jZW50b3MuaGRmcy5hbHSjggE0MIIBMKADAgESoQMCAQKiggEiBIIBHsmAmZRNC6tO7x6Iki0isqT+LJY43RLRt3DTbcmR07ykUEy+2ZY/hb3JltmdKlsi38/u2gvZ/5zAjJMUHM77SD6fwg2fwGvryHpYxCuDqZKzzuYZH0r2Tx/sLW/irks5wBWm4JiuUg9a+whxPfnrs05ncVQHLjR9dO9A9UKNabl7HkT9VtY6qrBvWvvVOhY53W21qsRZ/sCE+EREy1tL6wxp5JWgnCCN9IU4CC5cJLaiFxgCr5a3lAHZWVGfTS7KCG2IJAVeLd6VG+dNFSKI/5GPvegWZLYva5EY6AsH6KeNJr6RMavJ3r5Ml54kQjlosaByD07Vrp9dCQqbn6VXpEwdWzKaVG9I9B+Jnib1Rubfby9S6K/tajsX5ewL9XakgdswgdigAwIBEqKB0ASBzZLEg7uh8HC1V+SGB91vDQ6WFzhbMyp1d2YuZI6pSDJdclIFi61ViOzR7hvfSbqBNeNyK7eJJbkixtVstEcrVw8hqK4dtJsLuxUXYApx568qRxHFpnGAJsxp/oBQq3Hqw5JenV3JA3WawR25EKt/2qDG5cCBfJ9l/ct7FzxMak9xyNeiccMH3BpZ0sfITytP7wySlop9Ml5wFhs3deC3pyzxbx0MssKRXQT48jd/DqLOS1u8pefqQhTJ8wt00+zCbx3YSahVyTxfIHwewro=
> User-Agent: curl/7.81.0
> Accept: */*
> Content-Length: 204680523
> Expect: 100-continue
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
* Mark bundle as not supporting multiuse
< HTTP/1.1 307 Temporary Redirect
< Date: Tue, 11 Jan 2022 11:28:23 GMT
< Cache-Control: no-cache
< Expires: Tue, 11 Jan 2022 11:28:23 GMT
< Date: Tue, 11 Jan 2022 11:28:23 GMT
< Pragma: no-cache
< X-FRAME-OPTIONS: SAMEORIGIN
< WWW-Authenticate: Negotiate oYH1MIHyoAMKAQChCwYJKoZIhvcSAQICom4EbGBqBgkqhkiG9xIBAgICAG9bMFmgAwIBBaEDAgEPok0wS6ADAgESokQEQgzV9bwMFt51BNmSTC0kZ7Az9LyoleMLwa2aBXWmheNK8KU9SsWqoTJjlWOncJo2fxT703MLtsoV6FwObz8dNBnGyaNuBGxgagYJKoZIhvcSAQICAgBvWzBZoAMCAQWhAwIBD6JNMEugAwIBEqJEBEIM1fW8DBbedQTZkkwtJGewM/S8qJXjC8GtmgV1poXjSvClPUrFqqEyY5Vjp3CaNn8U+9NzC7bKFehcDm8/HTQZxsk=
< Set-Cookie: hadoop.auth="u=hdfs&p=hdfs/centos.hdfs.alt@HDFS.ALT&t=kerberos&e=1641936503003&s=4bunPVpKvdV0MYWPrmhaLPHKNABjEGcRA+1kqJ5w9XE="; Path=/; HttpOnly
* Rewind stream after send
< Location: http://centos.hdfs.alt:1006/webhdfs/v1/user/test_dir_alt/krb_spark-3.0.0.tar.gz5?op=CREATE&delegation=HQAEaGRmcwRoZGZzAIoBfkjlb92KAX5s8fPdj6kKFFs57jd12c5FMhHlYTCumaTghOBbEldFQkhERlMgZGVsZWdhdGlvbg4xMjcuMC4wLjE6OTAwMA&namenoderpcaddress=localhost:9000&createflag=&createparent=true&overwrite=false
< Content-Type: application/octet-stream
< Content-Length: 0
* Keep sending data to get tossed away!
<
* Send failure: Broken pipe
* Closing connection 0
curl: (55) Send failure: Broken pipe
I expected the following
Command ends without error
curl/libcurl version
curl 7.81.0 (x86_64-alt-linux-gnu) libcurl/7.81.0 OpenSSL/1.1.1l zlib/1.2.11 brotli/1.0.9 libidn2/2.3.2 libpsl/0.21.1 (+libidn2/2.3.1) nghttp2/1.41.0 OpenLDAP/2.4.59
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets
All versions of curl greater than 7.64.0 are affected by this issue.
operating system
The problem is present on all linux-based operating systems.