rgw: s3 bucket logging/server access logging by mikulely · Pull Request #14841 · ceph/ceph

mikulely · 2017-04-27T15:49:29Z

This is initial impl of s3 bucket logging/server access logging (
http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html )

We've discussed this on ML before (
https://www.spinics.net/lists/ceph-devel/msg35681.html )

Usage doc can be found at mikulely@dd52135

I've also writen a shell script as runbook to demonstrate BL usage can be found at
https://gist.github.com/mikulely/cb6b245c60fda4c1546aaa761c2e5615

We've rebased this functionality form Jewel to Master, and added test
cases to s3-tests in the following pr ceph/s3-tests#161

Current impl lack of Log Delivery Group ACL support
(http://docs.aws.amazon.com/AmazonS3/latest/dev/enable-logging-programming.html),
to someone who may concern.

~~potential issue/ future work~~

~~+ race condition when opslog add to and bl deliver from same opslog rdos-obj~~
~~+ cleanup those opslog rados-obj which opslog is enable but bl is disable~~

~~Those issue need future work to refactor current ops log impl.~~

Maybe we can rename opslog rados-obj when delivering, cause rados
object are located by name, rename means extra data copy.

We propose that we can organize opslogs into rados namespace,
namespace can be named by date(like 2017-04-47),and bl_deliver only
process namespace with older date.This process fit our deliver
scheduler(provide bucket logging for yesterday) and also avoiding race
condition. cleanup unused opslog also is very easy，just remove remaining
opslog objs in the namespace after delivery.

CC: @mattbenjamin @yehudasa @robbat2 (Plz take a test if you're still interested.)

Edited:

2017-06-23: strike through outdated description

robbat2 · 2017-04-27T16:44:42Z

@mikulely does this correctly handle load-balanced multiple RGW instances (eg haproxy in front of RGW)? I haven't read the entire series to see if it would, but wanted to ask anyway.

mattbenjamin · 2017-04-27T17:54:07Z

@mikulely from RGW bug scrub: do you have unit test support, or can provide?

mikulely · 2017-04-28T03:09:44Z

@robbat2 what does load-balanced multiple RGW instances mean? Let's clarify that the opslog record procedure is achieved by rgw_enble_ops_log impl , this pr only upload them to user bucket.

@mattbenjamin there's no unit test at function level, but we do write a bunch of python scripts for functionality test.

robbat2 · 2017-04-28T04:23:22Z

@mikulely The upload to user bucket, does it correctly take into account that multiple RGWs may be concurrently writing different entries in the ops log? Eg, during a multipart upload, individual parts may be concurrently uploaded to different RGWs.

mikulely · 2017-04-28T04:53:15Z

@robbat2 1. There‘s lock mechanism to guard only one rgw instance can handle certain bucket's opslog(copy from object lifecycle management impl )2. Every time rgw upload opslog to a unique rgw-object,the rgw-object name is generated by https://github.com/ceph/ceph/pull/14841/files#diff-ff97b7e16bb0ec2eef20f8068c88726cR158

mikulely · 2017-05-18T13:41:45Z

rebased with master

mikulely · 2017-06-06T02:48:24Z

Repushed and add LDG ACL support

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

eg: $ radosgw-admin log show { "bucket_id": "4a039cfd-ab25-408a-9b04-51b6983b4ed0.4134.1", "bucket_owner": "tester", "bucket": "swiftcontainer1", "log_entries": [ { "bucket": "swiftcontainer1", "time": "2017-09-07 12:20:34.903364Z", "time_local": "2017-09-07 20:20:34.903364", "remote_addr": "127.0.0.1", "user": "tester", "operation": "PUT", "uri": "/swift/v1/swiftcontainer1", "http_status": "201", "error_code": "Created", "bytes_sent": 0, "bytes_received": 0, "object_size": 0, "total_time": 157935, "user_agent": "python-swiftclient-3.4.0", "referrer": "", "prot_flags": "SWIFT_REST" } ], "log_sum": { "bytes_sent": 0, "bytes_received": 0, "total_time": 157935, "total_entries": 1 } } Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

track resource type via following endpoints: + s3 + s3website + swift auth + admin Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

Mgerge branch 'wip-support-operation' into wip-jiaying-bl rgw opslog: + did't recognise op issued via different endpoints，eg can‘t recognise get request from s3/s3website endpoint. + missing parsed http_method,(this one is not the same as the one that stored in the http request, which is a string) + didn't record resource_type,which need to be recorded during http routing this patch add three new fields: + prot_flag + http_method + resource [root@ceph-node1]~/jiaying/ceph/build# ../src/mrun j2 radosgw-admin log show --object=2017-09-15-20-2a6a2c76-d968-4c0f-800d-49c412fd61b5.4106.1-test { "bucket": "test", "time": "2017-09-15 12:26:49.704525Z", "time_local": "2017-09-15 20:26:49.704525", "remote_addr": "127.0.0.1", "user": "testid", "operation": "PUT", "uri": "/test/777777777", "http_status": "200", "error_code": "", "bytes_sent": 0, "bytes_received": 5242880, "object_size": 5242880, "total_time": 476762, "user_agent": "", "referrer": "", "prot_flags": "REST", "resource": "OBJECT", "http_method": "PUT" } [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/LogFormat.html

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

compile errno log: /home/mikulely/ceph/src/rgw/rgw_bl.cc: In function ‘std::string render_target_key(CephContext*, std::string)’: /home/mikulely/ceph/src/rgw/rgw_bl.cc:156:31: error: void value not ignored as it ought to be sizeof(unique_string_buf)); ^ /home/mikulely/ceph/src/rgw/rgw_bl.cc: In member function ‘int RGWBL::process()’: /home/mikulely/ceph/src/rgw/rgw_bl.cc:825:59: error: ‘get_random_bytes’ was not declared in this scope int ret = get_random_bytes((char *)&start, sizeof(start)); Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

mikulely · 2017-10-20T02:55:51Z

@yehudasa rebased against master：

fix compiler errno
merge operation field support ( pr rgw: opslog support record http_method/resource/prot_flag #17754)

Plz help to review.

stale · 2018-10-18T08:54:05Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
If you are a maintainer or core committer, please follow-up on this issue to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

mattbenjamin · 2018-10-18T11:30:43Z

unstale me my man

stale · 2018-12-17T16:57:34Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
If you are a maintainer or core committer, please follow-up on this issue to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

stale · 2019-02-15T17:22:39Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
If you are a maintainer or core committer, please follow-up on this issue to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

chuang-he · 2019-03-06T08:43:55Z

src/rgw/rgw_log.h

      ::decode(x_headers, p);
    }
+    if (struct_v >= 10) {
+      ::decode(prot_flags, p);


::decode(request_id, p); ?

stale · 2019-05-05T09:23:19Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

stale · 2019-08-03T09:34:25Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

mikulely mentioned this pull request Apr 27, 2017

rgw: S3 bucket logging #14825

Closed

liewegas added feature rgw labels Apr 27, 2017

mattbenjamin self-assigned this Apr 27, 2017

mikulely force-pushed the wip-jiaying-bl branch from 7f01511 to 264defb Compare April 28, 2017 02:35

mikulely force-pushed the wip-jiaying-bl branch from 264defb to 885fe82 Compare April 28, 2017 04:41

mikulely force-pushed the wip-jiaying-bl branch from 728ed24 to 4c13600 Compare May 10, 2017 06:32

mikulely force-pushed the wip-jiaying-bl branch 3 times, most recently from 612e4ba to 35bb385 Compare May 18, 2017 10:01

mikulely force-pushed the wip-jiaying-bl branch from 9b81095 to 31a22e6 Compare June 5, 2017 09:11

mikulely force-pushed the wip-jiaying-bl branch from 31a22e6 to 29fab09 Compare June 23, 2017 09:22

mikulely changed the title ~~rgw: impl s3 bucket logging/server access logging~~ rgw: s3 bucket logging/server access logging Jun 23, 2017

mikulely force-pushed the wip-jiaying-bl branch 10 times, most recently from ad0f05e to 4786ff2 Compare June 28, 2017 06:48

ZVampirEM77 and others added 12 commits October 18, 2017 15:44

rgw: cleanup req_state::prot_flags to RGW_REST_SWIFT_AUTH for swift_auth

347672a

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

rgw: make req_state::prot_flags support all rgw_enable_apis protocol

f77c6ec

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

rgw: track resource type during http route process

9fb3645

track resource type via following endpoints: + s3 + s3website + swift auth + admin Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

rgw: record parsed http_method into opslog entry

b8e7d66

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

rgw: bl make operation field s3 compatible

2b137b1

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

rgw: bl s/bl_deliver/bl-deliver/g

2bcf195

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

rgw: add _t suffix for http_method/resource/prot_flag map

cc44e0f

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

rgw: bl cleaup dout/debug message

ea2e6d2

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

rgw: bl adopt to new rgw_to_asctime()

fc6a4a9

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

mikulely force-pushed the wip-jiaying-bl branch from d96ea42 to fc6a4a9 Compare October 20, 2017 02:39

stale bot added the stale label Oct 18, 2018

stale bot removed the stale label Oct 18, 2018

stale bot added the stale label Dec 17, 2018

ceph-jenkins added the documentation label Dec 17, 2018

stale bot removed the stale label Dec 17, 2018

stale bot added the stale label Feb 15, 2019

chuang-he reviewed Mar 6, 2019

View reviewed changes

src/rgw/rgw_log.h

::decode(x_headers, p);

}

if (struct_v >= 10) {

::decode(prot_flags, p);

Copy link

chuang-he Mar 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

::decode(request_id, p); ?

stale bot removed the stale label Mar 6, 2019

stale bot added the stale label May 5, 2019

stale bot closed this Aug 3, 2019

Conversation

mikulely commented Apr 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robbat2 commented Apr 27, 2017

Uh oh!

mattbenjamin commented Apr 27, 2017

Uh oh!

mikulely commented Apr 28, 2017

Uh oh!

robbat2 commented Apr 28, 2017

Uh oh!

mikulely commented Apr 28, 2017

Uh oh!

mikulely commented May 18, 2017

Uh oh!

mikulely commented Jun 6, 2017

Uh oh!

mikulely commented Oct 20, 2017

Uh oh!

stale bot commented Oct 18, 2018

Uh oh!

mattbenjamin commented Oct 18, 2018

Uh oh!

stale bot commented Dec 17, 2018

Uh oh!

stale bot commented Feb 15, 2019

Uh oh!

chuang-he Mar 6, 2019

Choose a reason for hiding this comment

Uh oh!

stale bot commented May 5, 2019

Uh oh!

stale bot commented Aug 3, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

mikulely commented Apr 27, 2017 •

edited

Loading