rgw: opslog support record http_method/resource/prot_flag by mikulely · Pull Request #17754 · ceph/ceph

mikulely · 2017-09-15T12:49:52Z

Add three fields to support the Operation field[1] of s3 bucket
logging(#14841).

before

rgw opslog:

did't recognise op issued via different endpoints，eg can‘t recognise
get request from s3/s3website endpoint.
missing parsed http_method,(this one is not the same as the one that
stored in the http request, which is a string)
didn't record resource_type,which need to be recorded during http routing

after

we've add three new fields:

prot_flag
http_method
resource

[root@ceph-node1]~/jiaying/ceph/build# ../src/mrun j2 radosgw-admin  log show --object=2017-09-15-20-2a6a2c76-d968-4c0f-800d-49c412fd61b5.4106.1-test

       {
            "bucket": "test",
            "time": "2017-09-15 12:26:49.704525Z",
            "time_local": "2017-09-15 20:26:49.704525",
            "remote_addr": "127.0.0.1",
            "user": "testid",
            "operation": "PUT",
            "uri": "/test/777777777",
            "http_status": "200",
            "error_code": "",
            "bytes_sent": 0,
            "bytes_received": 5242880,
            "object_size": 5242880,
            "total_time": 476762,
            "user_agent": "",
            "referrer": "",
            "prot_flags": "REST",
            "resource": "OBJECT",
            "http_method": "PUT"
        }

[1] http://docs.aws.amazon.com/AmazonS3/latest/dev/LogFormat.html

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

eg: $ radosgw-admin log show { "bucket_id": "4a039cfd-ab25-408a-9b04-51b6983b4ed0.4134.1", "bucket_owner": "tester", "bucket": "swiftcontainer1", "log_entries": [ { "bucket": "swiftcontainer1", "time": "2017-09-07 12:20:34.903364Z", "time_local": "2017-09-07 20:20:34.903364", "remote_addr": "127.0.0.1", "user": "tester", "operation": "PUT", "uri": "/swift/v1/swiftcontainer1", "http_status": "201", "error_code": "Created", "bytes_sent": 0, "bytes_received": 0, "object_size": 0, "total_time": 157935, "user_agent": "python-swiftclient-3.4.0", "referrer": "", "prot_flags": "SWIFT_REST" } ], "log_sum": { "bytes_sent": 0, "bytes_received": 0, "total_time": 157935, "total_entries": 1 } } Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

track resource type via following endpoints: + s3 + s3website + swift auth + admin Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

yehudasa · 2017-11-17T17:30:50Z

src/rgw/rgw_rest.cc

  }

-  if (s->prot_flags & RGW_REST_SWIFT && ! quoted) {
+  if (s->prot_flags & (RGW_REST_SWIFT | RGW_REST_SWIFT_AUTH)


@ZVampirEM77 etag is not going to be sent on swift auth request.

yehudasa · 2017-11-17T17:33:32Z

src/rgw/rgw_rest.cc


-  if (s->prot_flags & RGW_REST_SWIFT && !content_type) {
+  if (s->prot_flags & (RGW_REST_SWIFT | RGW_REST_SWIFT_AUTH)
+      && !content_type) {


@ZVampirEM77 have you seen an actual issue here, or are you just pairing SWIFT_AUTH with SWIFT? I'm not sure this is needed, or won't break things somewhere.

yehudasa · 2017-11-17T18:05:03Z

src/rgw/rgw_log.cc

  formatter->dump_int("total_time", total_time);
  formatter->dump_string("user_agent",  entry.user_agent);
  formatter->dump_string("referrer",  entry.referrer);
+  formatter->dump_string("prot_flags", rgw_prot_flags[entry.prot_flags]);


@mikulely @ZVampirEM77 this is not going to work if flags have more than one bit set. Should generate a list of strings.

@yehudasa in what conditions will flags have more than one bit set? i think there is a one-to-one correlation between a request and its protocol flags.

yehudasa · 2017-11-17T18:05:31Z

src/rgw/rgw_log.h

+    ::encode(prot_flags, bl);
    ENCODE_FINISH(bl);
  }
  void decode(bufferlist::iterator &p) {


@ZVampirEM77 update version in decode() too

yehudasa · 2017-11-17T18:09:12Z

src/rgw/rgw_log.cc

  formatter->dump_string("user_agent",  entry.user_agent);
  formatter->dump_string("referrer",  entry.referrer);
  formatter->dump_string("prot_flags", rgw_prot_flags[entry.prot_flags]);
+  formatter->dump_string("resource", rgw_resources[entry.resource]);


@mikulely need to search for entry.resource and only dump if exists in rgw_resources, otherwise will create a new entry.

yehudasa · 2017-11-17T18:14:59Z

src/rgw/rgw_rest_metadata.cc

 }

 RGWOp *RGWHandler_Metadata::op_get() {
+  s->resource = RGW_RESOURCE_CATEGORY_METADATA;


there's a lot of duplicate stuff here, maybe can be moved into the constructor and when the handler is created you can pass in the req_state? Same goes to other handlers

yehudasa · 2017-11-17T18:18:15Z

src/rgw/rgw_log.cc

  formatter->dump_string("referrer",  entry.referrer);
  formatter->dump_string("prot_flags", rgw_prot_flags[entry.prot_flags]);
  formatter->dump_string("resource", rgw_resources[entry.resource]);
+  formatter->dump_string("http_method", rgw_http_methods[entry.http_method]);


@ZVampirEM77 how is this different from the original method string?

@yehudasa the original method string is like OP_GET etc which is not compatible with AWS S3 GET

yehudasa · 2017-11-17T18:19:30Z

src/rgw/rgw_log.h

  int prot_flags;
  headers_map x_headers;
  int resource;
+  int http_method;


missing initialization, where resource is initialized.

@yehudasa i think maybe you missed something. The entry. http_method is assigned in function rgw_log_op

stale · 2018-10-18T04:54:05Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
If you are a maintainer or core committer, please follow-up on this issue to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

stale · 2019-04-22T19:34:42Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

ZVampirEM77 added 2 commits September 15, 2017 19:26

rgw: cleanup req_state::prot_flags to RGW_REST_SWIFT_AUTH for swift_auth

bf8bc84

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

rgw: make req_state::prot_flags support all rgw_enable_apis protocol

82a4e9c

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

mikulely mentioned this pull request Sep 15, 2017

rgw: fix opslog operation field as per Amazon s3 #16984

Closed

ZVampirEM77 and others added 3 commits September 15, 2017 21:33

rgw: track resource type during http route process

1600f16

track resource type via following endpoints: + s3 + s3website + swift auth + admin Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

rgw: record parsed http_method into opslog entry

1e65f73

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

mikulely force-pushed the wip-support-operation branch from 9cbed34 to 1e65f73 Compare September 15, 2017 13:33

mikulely mentioned this pull request Oct 20, 2017

rgw: s3 bucket logging/server access logging #14841

Closed

liewegas added bug-fix rgw labels Nov 17, 2017

liewegas requested a review from yehudasa November 17, 2017 14:55

yehudasa reviewed Nov 17, 2017

View reviewed changes

stale bot added the stale label Oct 18, 2018

stale bot closed this Apr 22, 2019

yuvalif mentioned this pull request Sep 12, 2024

rgw: add support for GetBucketLogging and PutBucketLogging #54815

Closed

14 tasks

Conversation

mikulely commented Sep 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

before

after

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stale bot commented Oct 18, 2018

Uh oh!

stale bot commented Apr 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mikulely commented Sep 15, 2017 •

edited

Loading