Skip to content

Write deprecation logs to a data stream#61484

Merged
pugnascotia merged 6 commits intoelastic:masterfrom
pugnascotia:46106-index-deprecation-logs
Sep 3, 2020
Merged

Write deprecation logs to a data stream#61484
pugnascotia merged 6 commits intoelastic:masterfrom
pugnascotia:46106-index-deprecation-logs

Conversation

@pugnascotia
Copy link
Copy Markdown
Contributor

@pugnascotia pugnascotia commented Aug 24, 2020

Closes #46106. Implement a new log4j appender for deprecation logging, in order to write logs to a dedicated data stream. This is controlled by a new setting, cluster.deprecation_indexing.enabled.

Depends on #61474.

Test by running ./gradlew run and then:

AUTH="elastic-admin:elastic-password"

# Enable new setting
curl -u $AUTH -XPUT \
  --data '{ "transient": { "cluster.deprecation_indexing.enabled": true } }' \
  http://localhost:9200/_cluster/settings 

# Trigger deprecation warning
curl -u $AUTH http://localhost:9200/_flush/synced?pretty

# Show indexed deprecation messages
curl -u $AUTH http://localhost:9200/logs-deprecation-elasticsearch/_search?pretty

@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented Aug 24, 2020

Pinging @elastic/es-core-infra (:Core/Infra/Logging)

@elasticmachine elasticmachine added the Team:Core/Infra Meta label for core/infra team label Aug 24, 2020
@pugnascotia pugnascotia mentioned this pull request Aug 24, 2020
Closed
@pugnascotia pugnascotia force-pushed the 46106-index-deprecation-logs branch from 9125b65 to 126f0ca Compare August 25, 2020 10:28
@pugnascotia
Copy link
Copy Markdown
Contributor Author

pugnascotia commented Sep 1, 2020

@elasticmachine run elasticsearch-ci/packaging-sample-windows

jakelandis
jakelandis reviewed Sep 1, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@jakelandis jakelandis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good. a couple of comments and a PR to this PR pugnascotia#3 (to ensure using the latest testing plugins)

@pugnascotia
Merge pull request #3 from jakelandis/46106-index-deprecation-logs-jake
035bd6e 
update gradle config for new testing plugins
jakelandis
jakelandis approved these changes Sep 2, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@jakelandis jakelandis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested locally and worked great! LGTM

pugnascotia
pugnascotia commented Sep 2, 2020
View reviewed changes
server/src/main/java/org/elasticsearch/common/logging/DeprecatedMessage.java
.field("data_stream.type", "logs")
.field("data_stream.datatype", "deprecation")
.field("data_stream.namespace", "elasticsearch")
.field("ecs.version", ECS_VERSION)
Copy link
Copy Markdown
Contributor Author

@pugnascotia pugnascotia Sep 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jakelandis how do these look to you?

Copy link
Copy Markdown
Contributor

@jakelandis jakelandis Sep 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

pgomulka
pgomulka approved these changes Sep 3, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@pgomulka pgomulka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM,
we would need to update beats test samples (with new fields). I can help with this.

@pugnascotia pugnascotia merged commit dce2ef9 into elastic:master Sep 3, 2020
@pugnascotia pugnascotia deleted the 46106-index-deprecation-logs branch September 3, 2020 13:50
pugnascotia added a commit to pugnascotia/elasticsearch that referenced this pull request Sep 4, 2020
@pugnascotia
Write deprecation logs to a data stream
441aef1 
Backport of elastic#61484.

Closes elastic#46106. Implement a new log4j appender for deprecation logging, in
order to write logs to a dedicated data stream. This is controlled by a new
setting, `cluster.deprecation_indexing.enabled`.
ruflin
ruflin reviewed Sep 24, 2020
View reviewed changes
server/src/main/java/org/elasticsearch/common/logging/DeprecatedMessage.java

return new ESLogMessage(messagePattern, args)
.field("data_stream.type", "logs")
.field("data_stream.datatype", "deprecation")
Copy link
Copy Markdown
Contributor

@ruflin ruflin Sep 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pugnascotia This should be data_stream.dataset to be aligned with the indexing strategy.

I would also propose to keep the namespace as default and use deprecation.elasticsearch as the dataset name. Only important thing is that the dataset does not contain a -.

Copy link
Copy Markdown
Contributor Author

@pugnascotia pugnascotia Sep 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ruflin so would we have the following, then?

    .field("data_stream.dataset", "default")
    .field("data_stream.namespace", "deprecation.elasticsearch")

Copy link
Copy Markdown
Contributor

@ruflin ruflin Sep 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the other way around:

.field("data_stream.dataset", "deprecation.elasticsearch")
.field("data_stream.namespace", "default")

Copy link
Copy Markdown
Contributor Author

@pugnascotia pugnascotia Sep 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I'll get that changed 👍

@pugnascotia pugnascotia mentioned this pull request Sep 24, 2020
Merged
pugnascotia added a commit that referenced this pull request Sep 24, 2020
@pugnascotia
Tweak the ECS fields in DeprecatedMessage (#62855)
d0c0ca1 
Follow-up to #61484.
pugnascotia added a commit that referenced this pull request Sep 24, 2020
@pugnascotia
Tweak the ECS fields in DeprecatedMessage (#62855)
7771d8b 
Backport of #62855. Follow-up to #61484.
@pugnascotia pugnascotia mentioned this pull request Oct 19, 2020
Closed
@Mpdreamz Mpdreamz mentioned this pull request Nov 16, 2020
Closed
61 tasks
@stevejgordon stevejgordon mentioned this pull request Dec 17, 2020
Closed
@alisonelizabeth alisonelizabeth mentioned this pull request Aug 10, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ruflin ruflin ruflin left review comments

@jakelandis jakelandis jakelandis approved these changes

@rjernst rjernst Awaiting requested review from rjernst

+1 more reviewer

@pgomulka pgomulka pgomulka approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

:Core/Infra/Logging Log management and logging utilities >enhancement Team:Core/Infra Meta label for core/infra team v7.10.0 v8.0.0-alpha1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Write deprecation logs to an index

5 participants

@pugnascotia @elasticmachine @ruflin @jakelandis @pgomulka