Skip to content

Deprecate index audit output type#37671

Merged
albertzaharovits merged 10 commits intoelastic:6.xfrom
albertzaharovits:deprecate_index_audit_6x
Jan 24, 2019
Merged

Deprecate index audit output type#37671
albertzaharovits merged 10 commits intoelastic:6.xfrom
albertzaharovits:deprecate_index_audit_6x

Conversation

@albertzaharovits
Copy link
Copy Markdown
Contributor

@albertzaharovits albertzaharovits commented Jan 21, 2019

Supersedes #37301

This PR deprecates the index audit output.
In general, the problem with the index audit output is that event indexing can be slower than the rate with which audit events are generated, especially during the daily rollovers or the rolling cluster upgrades. In this situation audit events will be lost which is a terrible failure case for an audit system.

I will follow-up with the removal PR for 7.0 .

EDIT:
Besides of the settings under the xpack.security.audit.index namespace, the xpack.security.audit.outputs setting has also been deprecated and will be removed in 7. Although explicitly configuring the logfile output does not touch any deprecation bits, this setting is made redundant in 7 so this PR deprecates it as well.

Relates #29881

@elasticmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-security

Copy link
Copy Markdown
Contributor

@lcawl lcawl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

static DeprecationIssue auditIndexSettingsCheck(List<NodeInfo> nodeInfos, List<NodeStats> nodeStats) {
List<String> nodesFound = nodeInfos.stream()
.filter(nodeInfo -> (nodeInfo.getSettings().getByPrefix("xpack.security.audit.outputs").isEmpty() == false)
|| (nodeInfo.getSettings().getByPrefix("xpack.security.audit.index").isEmpty() == false))
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to produce a deprecation issue if the outputs is non-empty, even if it only contains logfile.
Is that what you intended? It seems strange to deprecate any explicit configuration of the output type.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is intended. I plan to remove this setting altogether in 7.
There is no point of having a setting for the output type when there is only one output type.
The deprecation is there to warn that the setting will go away.

? Collections.emptyList()
: Collections.singletonList(LoggingAuditTrail.NAME),
Property.NodeScope);
Property.NodeScope, Property.Deprecated);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above, is this really what we want?
I'm not totally against it, but the description of the PR, and the docs changes don't indicate that we're deprecating the outputs setting. Since there's only 1 non-deprecated value, and that's the default it makes some sense to just say the setting is obsolete, but the rest of this PR doesn't say that.

Copy link
Copy Markdown
Contributor Author

@albertzaharovits albertzaharovits Jan 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I will update the description. Sorry if it came across as a "hidden" change!

Copy link
Copy Markdown
Member

@jaymode jaymode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Let's give @tvernum a chance to take another look before merging.

cluster or a separate cluster.
cluster or a separate cluster. deprecated[6.7.0, `logfile` will be the only audit
option available in 7.0. A separate log file ingestion component should instead
be used for indexing the audit trail.]
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This deprecation comment confused me on my earlier review, because the text talks about "logfile" but not "output".
It probably reads OK in context, but I would have phrased it more like:

The outputs setting will be removed in 7.0 as there will only be one supported output type (logfile). Users who wish to store their audit information in an Elasticsearch index should write to the file output, and a use a file ingestion component to index it into Elasticsearch.

Entirely up to you whether you want to update it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I have made the change as suggested! thanks 👍

@albertzaharovits
Copy link
Copy Markdown
Contributor Author

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/5755/console

10:31:26 ERROR 40.1s J1 | SnapshotDisruptionIT.testDisruptionOnSnapshotInitialization <<< FAILURES!
10:31:26 > Throwable #1: SnapshotException[[test-repo:test-snap-2/Ps0flXe1TIO-UEb1CmubAQ] Snapshot could not be read]; nested: SnapshotMissingException[[test-repo:test-snap-2/Ps0flXe1TIO-UEb1CmubAQ] is missing]; nested: NoSuchFileException[/var/lib/jenkins/workspace/elastic+elasticsearch+pull-request-1/server/build/testrun/integTest/J1/temp/org.elasticsearch.discovery.SnapshotDisruptionIT_BDC0C1E1ED934767-001/tempDir-002/repos/kzYQFjRhEu/snap-Ps0flXe1TIO-UEb1CmubAQ.dat];
10:31:26 > at __randomizedtesting.SeedInfo.seed([BDC0C1E1ED934767:971FE731CB8BB4B8]:0)
10:31:26 > at org.elasticsearch.snapshots.SnapshotsService.snapshots(SnapshotsService.java:204)
10:31:26 > at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:136)
10:31:26 > at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:55)
10:31:26 > at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:124)
10:31:26 > at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:211)
10:31:26 > at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
10:31:26 > at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
10:31:26 > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
10:31:26 > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
10:31:26 > at java.lang.Thread.run(Thread.java:748)
10:31:26 > Caused by: SnapshotMissingException[[test-repo:test-snap-2/Ps0flXe1TIO-UEb1CmubAQ] is missing]; nested: NoSuchFileException[/var/lib/jenkins/workspace/elastic+elasticsearch+pull-request-1/server/build/testrun/integTest/J1/temp/org.elasticsearch.discovery.SnapshotDisruptionIT_BDC0C1E1ED934767-001/tempDir-002/repos/kzYQFjRhEu/snap-Ps0flXe1TIO-UEb1CmubAQ.dat];
10:31:26 > at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getSnapshotInfo(BlobStoreRepository.java:557)
10:31:26 > at org.elasticsearch.snapshots.SnapshotsService.snapshots(SnapshotsService.java:198)
10:31:26 > ... 9 more
10:31:26 > Caused by: java.nio.file.NoSuchFileException: /var/lib/jenkins/workspace/elastic+elasticsearch+pull-request-1/server/build/testrun/integTest/J1/temp/org.elasticsearch.discovery.SnapshotDisruptionIT_BDC0C1E1ED934767-001/tempDir-002/repos/kzYQFjRhEu/snap-Ps0flXe1TIO-UEb1CmubAQ.dat
10:31:26 > at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
10:31:26 > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
10:31:26 > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
10:31:26 > at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
10:31:26 > at java.nio.file.Files.newByteChannel(Files.java:361)
10:31:26 > at java.nio.file.Files.newByteChannel(Files.java:407)
10:31:26 > at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
10:31:26 > at org.apache.lucene.mockfile.FilterFileSystemProvider.newInputStream(FilterFileSystemProvider.java:192)
10:31:26 > at org.apache.lucene.mockfile.FilterFileSystemProvider.newInputStream(FilterFileSystemProvider.java:192)
10:31:26 > at org.apache.lucene.mockfile.FilterFileSystemProvider.newInputStream(FilterFileSystemProvider.java:192)
10:31:26 > at org.apache.lucene.mockfile.HandleTrackingFS.newInputStream(HandleTrackingFS.java:92)
10:31:26 > at org.apache.lucene.mockfile.FilterFileSystemProvider.newInputStream(FilterFileSystemProvider.java:192)
10:31:26 > at org.apache.lucene.mockfile.HandleTrackingFS.newInputStream(HandleTrackingFS.java:92)
10:31:26 > at java.nio.file.Files.newInputStream(Files.java:152)
10:31:26 > at org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:120)
10:31:26 > at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.readBlob(ChecksumBlobStoreFormat.java:101)
10:31:26 > at org.elasticsearch.repositories.blobstore.BlobStoreFormat.read(BlobStoreFormat.java:90)
10:31:26 > at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getSnapshotInfo(BlobStoreRepository.java:555)
10:31:26 > ... 10 more
10:31:26 Completed [264/331] on J1 in 40.10s, 1 test, 1 error <<< FAILURES!

@elasticmachine run elasticsearch-ci/1

@albertzaharovits albertzaharovits merged commit 8765a31 into elastic:6.x Jan 24, 2019
@albertzaharovits albertzaharovits deleted the deprecate_index_audit_6x branch January 24, 2019 10:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants