Skip to content

sei: multiple packages with duplicated field definitions #4398

@efd6

Description

@efd6

While preparing packages for the ECS updates I found that duplicated field definitions now caused indeterminate outcomes from elastic-package build. This prevented tests from passing while doing the updates and so I fixed the packages that were failing making use of new elastic-package behaviour for vetting v2 packages for duplicate field definitions. At the time, elastic-package did not continue to vet packages beyond failures in the manifest checks, which meant that it was necessary to fix those before being able to investigate duplications.

elastic-package now proceeds to check duplications even when there are manifest vet failures (here and here), and it has become clear that the packages that I found in the original pass based on test failures did not find all cases (for example google_workspace fixed here).

So here is a complete list of non-deprecated SEI packages that have duplicated field definitions found using the script at the footer of this issue. The script requires elastic-package v0.65.0 and was run on the tree at 28fecbd from the packages directory.

for p in *; do
	grep 'elastic/security-external-integrations' ${p}/manifest.yml >/dev/null || continue
	grep '^description: Deprecated' ${p}/manifest.yml >/dev/null && continue
	gsed -i -e 's/^format_version: 1.0.0/format_version: 2.0.0/' -e '/^license: .*/d' ${p}/manifest.yml;
	(
		cd $p
		m="$(elastic-package build 2>&1 | grep 'defined multiple')"
		if [ "$m" != "" ]; then
			echo "- [ ] $p"
			echo $m \
			| gsed -r 's|^ +[0-9]+\. field "(.*)" is defined multiple times for data stream (.*), found in:|    \2: `\1`|g' \
			| gsed -r 's|/[^ ]+/data_stream/[^ ]+/fields/||g' \
			| sort
		fi
	)
	git reset --hard >/dev/null
done

Metadata

Metadata

Assignees

No one assigned

    Labels

    Team:Cloud-MonitoringLabel for the Cloud Monitoring teambugSomething isn't working, use only for issuesintegrationLabel used for meta issues tracking each integration

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions