Skip to content

[Feature-13419][Remote Logging] Add support for writing task logs to S3#13649

Merged
EricGao888 merged 4 commits intoapache:devfrom
rickchengx:Feature-13419
Mar 13, 2023
Merged

[Feature-13419][Remote Logging] Add support for writing task logs to S3#13649
EricGao888 merged 4 commits intoapache:devfrom
rickchengx:Feature-13419

Conversation

@rickchengx
Copy link
Copy Markdown
Contributor

@rickchengx rickchengx commented Mar 1, 2023

Purpose of the pull request

Brief change log

Add support for writing task logs to S3

Verify this pull request

UT

@rickchengx rickchengx mentioned this pull request Mar 1, 2023
Closed
3 tasks
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Mar 1, 2023 •
edited
Loading

Codecov Report

Merging #13649 (3792ec6) into dev (43d79e4) will decrease coverage by 0.10%.
The diff coverage is 18.80%.

❗ Current head 3792ec6 differs from pull request most recent head 89db373. Consider uploading reports for the commit 89db373 to get more accurate results

@@             Coverage Diff              @@
##                dev   #13649      +/-   ##
============================================
- Coverage     39.08%   38.98%   -0.10%     
+ Complexity     4426     4413      -13     
============================================
  Files          1128     1128              
  Lines         42111    42130      +19     
  Branches       4775     4776       +1     
============================================
- Hits          16458    16426      -32     
- Misses        23837    23893      +56     
+ Partials       1816     1811       -5
Impacted Files Coverage Δ
...che/dolphinscheduler/api/python/PythonGateway.java 17.24% <0.00%> (-0.23%) ⬇️
...heduler/api/service/impl/ResourcesServiceImpl.java 38.45% <ø> (-0.06%) ⬇️
...e/dolphinscheduler/common/constants/Constants.java 75.00% <ø> (ø)
...cheduler/common/constants/DataSourceConstants.java 0.00% <ø> (ø)
...heduler/common/log/remote/OssRemoteLogHandler.java 0.00% <0.00%> (-16.00%) ⬇️
...ler/common/log/remote/RemoteLogHandlerFactory.java 0.00% <0.00%> (ø)
...nscheduler/common/log/remote/RemoteLogService.java 0.00% <0.00%> (ø)
...cheduler/common/log/remote/S3RemoteLogHandler.java 0.00% <0.00%> (ø)
...r/server/master/processor/StateEventProcessor.java 0.00% <0.00%> (ø)
...er/server/master/processor/TaskEventProcessor.java 0.00% <0.00%> (ø)
... and 63 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@Radeity
Copy link
Copy Markdown
Member

Radeity commented Mar 1, 2023 •
edited
Loading

Hi, @rickchengx , may i ask why we have to create new instance of RemoteLogHandler to send remote log every time, can we just create a single instance to handle that?

@rickchengx rickchengx marked this pull request as draft March 1, 2023 12:52
@rickchengx
Copy link
Copy Markdown
Contributor Author

rickchengx commented Mar 1, 2023 •
edited
Loading

Hi, @rickchengx , may i ask why we have to create new instance of RemoteLogHandler to send remote log every time, can we just create a single instance to handle that?

Hi, @Radeity Thanks a lot for your comment. I've changed to singleton pattern.

@rickchengx rickchengx marked this pull request as ready for review March 2, 2023 03:38
@caishunfeng caishunfeng added feature new feature 3.2.0 labels Mar 6, 2023
rickchengx and others added 2 commits March 8, 2023 10:31
@rickchengx @caishunfeng
Update dolphinscheduler-common/src/main/java/org/apache/dolphinschedu…
6782463 
…ler/common/log/remote/S3RemoteLogHandler.java

Co-authored-by: caishunfeng <caishunfeng2021@gmail.com>
@rickchengx
use try with resource
89db373
@rickchengx rickchengx closed this Mar 8, 2023
@rickchengx rickchengx reopened this Mar 8, 2023
@rickchengx
Copy link
Copy Markdown
Contributor Author

rickchengx commented Mar 8, 2023

Hi, @caishunfeng , thanks a lot for your comment. I've changed it according to your suggestions and please help review this again.

EricGao888
EricGao888 previously approved these changes Mar 10, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@EricGao888 EricGao888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would u like to double check when available? @caishunfeng @Radeity

Radeity
Radeity previously approved these changes Mar 10, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@Radeity Radeity left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@rickchengx rickchengx dismissed stale reviews from Radeity and EricGao888 via fba7714 March 10, 2023 09:59
@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Mar 10, 2023

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 3 Code Smells

6.8% 6.8% Coverage
0.0% 0.0% Duplication

@EricGao888 EricGao888 merged commit aa79c01 into apache:dev Mar 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@caishunfeng caishunfeng caishunfeng left review comments

@EricGao888 EricGao888 EricGao888 approved these changes

@Radeity Radeity Radeity left review comments

@kezhenxu94 kezhenxu94 Awaiting requested review from kezhenxu94

@zhongjiajie zhongjiajie Awaiting requested review from zhongjiajie

@Tianqi-Dotes Tianqi-Dotes Awaiting requested review from Tianqi-Dotes

Assignees

@rickchengx rickchengx

Labels

backend document feature new feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature][Remote Logging] Add support for writing task logs to S3

5 participants

@rickchengx @codecov-commenter @Radeity @caishunfeng @EricGao888