Update runc and containerd by jsoriano · Pull Request #28621 · elastic/beats

jsoriano · 2021-10-25T13:49:04Z

What does this PR do?

Update runc and contained dependencies.

Why is it important?

Solve some dependabot warnings (the issues are starting containers and pulling images, something beats don't do).
runc 1.0 has been finally released, depend on it.

Checklist

My code follows the style guidelines of this project
~~I have commented my code, particularly in hard-to-understand areas~~
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
~~I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

elasticmachine · 2021-10-25T13:49:07Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-10-25T13:49:07Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

jsoriano · 2021-10-25T13:49:16Z

/package

elasticmachine · 2021-10-25T14:14:10Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-10-27T20:20:47.312+0000
Duration: 128 min 41 sec
Commit: 2fe5c88

Test stats 🧪

Test	Results
Failed	0
Passed	328
Skipped	31
Total	359

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

jsoriano · 2021-10-25T14:35:49Z

Updating docker to try to solve issue with dockerlogbeat build:

[2021-10-25T14:07:07.814Z] + mage packageSystemTests
[2021-10-25T14:07:09.720Z] # github.com/docker/docker/pkg/archive
[2021-10-25T14:07:09.720Z] ../../../../../../pkg/mod/github.com/docker/engine@v0.0.0-20191113042239-ea84732a7725/pkg/archive/archive_unix.go:84:5: undefined: "github.com/opencontainers/runc/libcontainer/system".RunningInUserNS
[2021-10-25T14:07:09.720Z] Error: error compiling magefiles
script returned exit code 1

ChrsMark

The change looks sane to me if CI agrees :).

ChrsMark · 2021-10-25T14:57:53Z

It will also cover #28464 which has some CI issues.

jlind23

LGTM

ChrsMark · 2021-10-26T09:33:51Z

/test

jsoriano · 2021-10-26T09:39:11Z

The issue with the generator is legit, I can reproduce it locally, I am trying to find what happens there.

jsoriano · 2021-10-26T10:54:50Z

Ok, the problem is that the generator uses the replaces from the go.mod in the current working copy, but the beats code from upstream master. In CI this means that it uses the replaces from the branch in the PR, but the code from master. This makes some changes in the replaces complicated without breaking the community beats. In this PR I am updating docker and removing the replace, I will try to do the change without needing to change this replace, but not sure if possible.

Here it uses the current working copy: https://github.com/elastic/beats/blob/16aa13e342609adad69cc092d3649982da71ddcd/generator/common/beatgen/setup/setup.go#L48

But the code from the current upstream master:
https://github.com/elastic/beats/blob/16aa13e342609adad69cc092d3649982da71ddcd/generator/common/Makefile#L40
https://github.com/elastic/beats/blob/16aa13e342609adad69cc092d3649982da71ddcd/generator/common/beatgen/beatgen.go#L159-L166

This reverts commit 16aa13e342609adad69cc092d3649982da71ddcd.

This reverts commit 3903c0282caca4282be2a815b3d8d96fbc4bbb6e.

jsoriano · 2021-10-26T17:27:25Z

x-pack/dockerlogbeat/magefile.go

-		IncludeFiles: []string{"rootfs", "config.json"},
-	}
-	archive, err := archive.TarWithOptions(buildDir, archiveOpts)
+	archive, err := tar(buildDir, "rootfs", "config.json")


@fearful-symmetry could you please take a look to the changes here?

I am doing this because updating containerd broke github.com/docker/docker/pkg/archive. The solution would be to upgrade docker, but upgrading docker breaks community beats (more info in other comment).

I saw that we are already using the tar command in other parts of this magefile, so I guess it is ok to use it here too 🙂 I gave it a try and it seems to work. Could you please double-check?

As a side advantage, after removing github.com/docker/docker/pkg/archive from here, runc doesn't appear anymore in go.mod.

We can also decide to update docker and break (or remove) community beats in master/8.0, but I would leave this discussion for later.

Gimme a few, lemme pull this down and poke around.

So, the build seems to work, I suspect I just did that because I'm not a fan of shelling out when I don't need to.

Thanks for checking!

I'm not a fan of shelling out when I don't need to.

+1 to this 🙂 But in this case we remove a problematic dependency, and we are already shelling out for other things. If this becomes problematic we can look for a different tar library in the future.

jsoriano · 2021-10-27T08:08:14Z

There were additional changes since last review, could I have another round? 🙂

jsoriano · 2021-10-27T08:10:25Z

E2E test failures are not related, they are also failing in master.

mergify · 2021-10-27T16:04:53Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b dependabot-runc-containerd upstream/dependabot-runc-containerd
git merge upstream/master
git push upstream dependabot-runc-containerd

…tainerd

jsoriano · 2021-10-27T20:20:03Z

/test

jsoriano · 2021-10-28T10:01:15Z

E2E failures are unrelated.

Solve some dependabot warnings (the issues are starting containers and pulling images, something beats don't do). Removes a dependency on runc in dockerlogbeat build scripts. (cherry picked from commit 97818c3) # Conflicts: # NOTICE.txt # go.mod # go.sum

Solve some dependabot warnings (the issues are starting containers and pulling images, something beats don't do). Removes a dependency on runc in dockerlogbeat build scripts. (cherry picked from commit 97818c3) # Conflicts: # go.sum

Solve some dependabot warnings (the issues are starting containers and pulling images, something beats don't do). Removes a dependency on runc in dockerlogbeat build scripts. (cherry picked from commit 97818c3) Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

Solve some dependabot warnings (the issues are starting containers and pulling images, something beats don't do). Removes a dependency on runc in dockerlogbeat build scripts.

jsoriano added Team:Integrations Label for the Integrations team backport-v7.16.0 Automated backport with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Oct 25, 2021

jsoriano requested review from a team and ChrsMark October 25, 2021 13:49

jsoriano self-assigned this Oct 25, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 25, 2021

ChrsMark approved these changes Oct 25, 2021

View reviewed changes

jlind23 approved these changes Oct 25, 2021

View reviewed changes

jsoriano added 6 commits October 26, 2021 12:57

Update runc and containerd

cb900e1

Update docker engine

892eda1

Additional argument in ContainerCreate

d55b547

Revert "Additional argument in ContainerCreate"

1e90bd9

This reverts commit 16aa13e342609adad69cc092d3649982da71ddcd.

Revert "Update docker engine"

a7b7c7c

This reverts commit 3903c0282caca4282be2a815b3d8d96fbc4bbb6e.

Try with containerd 1.4.11

871a199

jsoriano force-pushed the dependabot-runc-containerd branch from 16aa13e to 871a199 Compare October 26, 2021 11:12

jsoriano added 2 commits October 26, 2021 15:27

Use tar instead of docker archiver

0bab6ee

Try agin with latest containerd

5868526

jsoriano marked this pull request as draft October 26, 2021 14:04

jsoriano marked this pull request as ready for review October 26, 2021 17:18

jsoriano commented Oct 26, 2021

View reviewed changes

jsoriano requested review from ChrsMark, fearful-symmetry and jlind23 October 26, 2021 17:27

fearful-symmetry approved these changes Oct 27, 2021

View reviewed changes

Merge remote-tracking branch 'origin/master' into dependabot-runc-con…

2fe5c88

…tainerd

jsoriano merged commit 97818c3 into elastic:master Oct 28, 2021

jsoriano deleted the dependabot-runc-containerd branch October 28, 2021 10:02

jsoriano added the backport-v8.0.0 Automated backport with mergify label Oct 28, 2021

mergify bot mentioned this pull request Oct 28, 2021

[7.16](backport #28621) Update runc and containerd #28693

Merged

mergify bot mentioned this pull request Oct 28, 2021

[8.0](backport #28621) Update runc and containerd #28694

Merged

ChrsMark mentioned this pull request Oct 29, 2021

Upgrade github.com/Microsoft/go-winio #28464

Closed

This was referenced Oct 29, 2021

Upgrade prometheus library #28716

Merged

Fix dependency issues with rpm package #28720

Closed

Conversation

jsoriano commented Oct 25, 2021

What does this PR do?

Why is it important?

Checklist

Uh oh!

elasticmachine commented Oct 25, 2021

Uh oh!

elasticmachine commented Oct 25, 2021

Uh oh!

jsoriano commented Oct 25, 2021

Uh oh!

elasticmachine commented Oct 25, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

jsoriano commented Oct 25, 2021

Uh oh!

ChrsMark left a comment

Choose a reason for hiding this comment

Uh oh!

ChrsMark commented Oct 25, 2021

Uh oh!

jlind23 left a comment

Choose a reason for hiding this comment

Uh oh!

ChrsMark commented Oct 26, 2021

Uh oh!

jsoriano commented Oct 26, 2021

Uh oh!

jsoriano commented Oct 26, 2021

Uh oh!

jsoriano Oct 26, 2021

Choose a reason for hiding this comment

Uh oh!

jsoriano Oct 26, 2021

Choose a reason for hiding this comment

Uh oh!

fearful-symmetry Oct 26, 2021

Choose a reason for hiding this comment

Uh oh!

fearful-symmetry Oct 26, 2021

Choose a reason for hiding this comment

Uh oh!

jsoriano Oct 27, 2021

Choose a reason for hiding this comment

Uh oh!

jsoriano commented Oct 27, 2021

Uh oh!

jsoriano commented Oct 27, 2021

Uh oh!

mergify bot commented Oct 27, 2021

Uh oh!

jsoriano commented Oct 27, 2021

Uh oh!

jsoriano commented Oct 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

elasticmachine commented Oct 25, 2021 •

edited by jenkins-beats-ci bot

Loading