Ceph » rgw

After trying to upload an object to a non-existing S3-Bucket via PutObject (which returns a 404 Not Found with error code NoSuchBucket as expected) any subsequent request on the same HTTP connection fails.

Depending on the length of the PutObject request the subsequent request

• results in a 400 Bad Request error (with log message "failed to read header: bad method") for small files or
• a read timeout on the connection is encountered by the client for larger files

It looks like the failed PutObject request causes the HTTP parsing to fail afterwards.
(It seems that rgw is trying to read the request body of the failed PutObject request before parsing the next request. However the body is never sent by the client because instead of the expected "100 Continue" status response the sever sent the 404 status.)

The following python script reproces the issue and results in a 400 Bad Request error:

#!/usr/bin/python

import boto3

s3_endpoint_url = "" 
s3_access_key_id = "" 
s3_secret_access_key = "" 

s3 = boto3.resource('s3',
    '',
    use_ssl = False,
    verify = False,
    endpoint_url = s3_endpoint_url,
    aws_access_key_id = s3_access_key_id,
    aws_secret_access_key = s3_secret_access_key,
)

try:
    s3.meta.client.put_object(Bucket='foo', Key='bar', Body='body')
except s3.meta.client.exceptions.NoSuchBucket:
    pass

s3.meta.client.create_bucket(Bucket='foo')

Output:

Traceback (most recent call last):
  File "/tmp/badMethod.py", line 23, in <module>
    s3.meta.client.create_bucket(Bucket='foo')
  File "/usr/lib/python3.10/site-packages/botocore/client.py", line 514, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3.10/site-packages/botocore/client.py", line 938, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the CreateBucket operation: Bad Request

from the rgw logs:

1 failed to read header: bad method
1 ====== req done http_status=400 ======

ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)