Skip to content

Replace SetEnv invocation by base.Env to unlock parallelization#3111

Merged
AkihiroSuda merged 1 commit intocontainerd:mainfrom
apostasie:dev-3081
Jun 18, 2024
Merged

Replace SetEnv invocation by base.Env to unlock parallelization#3111
AkihiroSuda merged 1 commit intocontainerd:mainfrom
apostasie:dev-3081

Conversation

@apostasie
Copy link
Copy Markdown
Contributor

This is to fix #3081

cosign related code (which required signature change) and related tests have been cleaned-up slightly on top of that and parallelized.

Copy link
Copy Markdown
Member

@AkihiroSuda AkihiroSuda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@AkihiroSuda AkihiroSuda added this to the v2.0.0 milestone Jun 18, 2024
@AkihiroSuda AkihiroSuda added area/ci e.g., CI failure area/cosign cosign labels Jun 18, 2024
@apostasie
Copy link
Copy Markdown
Contributor Author

apostasie commented Jun 18, 2024

There is something wrong will github networking right now.
I/O timeouts aplenty and IPFS test seems to be down (<- this one probably on me).

Will force push & poke the CI again later.

@AkihiroSuda
Copy link
Copy Markdown
Member

=== RUN   TestComposePushAndPullWithCosignVerify
    testutil.go:567: buildkitHost="unix:///run/buildkit-nerdctl-test/buildkitd.sock"
=== PAUSE TestComposePushAndPullWithCosignVerify
=== CONT  TestComposePushAndPullWithCosignVerify
    compose_run_linux_test.go:486: projectName="nerdctl-compose-test689402566"
    compose_run_linux_test.go:490: assertion failed: res.ExitCode is not exitCode: panic: runtime error: index out of range [1] with length 1
        
        goroutine 1 [running]:
        github.com/compose-spec/compose-go/utils.GetAsEqualsMap({0xc0008d49c0, 0xd, 0x30?})
        	/go/pkg/mod/github.com/compose-spec/compose-go@v1.20.2/utils/stringutils.go:46 +0xfd
        github.com/compose-spec/compose-go/cli.WithOsEnv(0xc00050c380)
        	/go/pkg/mod/github.com/compose-spec/compose-go@v1.20.2/cli/options.go:231 +0x2c
        github.com/compose-spec/compose-go/cli.NewProjectOptions({0xc0002e00e0, 0x1, 0x1}, {0xc0008d2660, 0x6, 0x10?})
        	/go/pkg/mod/github.com/compose-spec/compose-go@v1.20.2/cli/options.go:93 +0xcf
        github.com/containerd/nerdctl/v2/pkg/composer.New({{0x0, 0x0}, {0x0, 0x0}, {0xc0002e00e0, 0x1, 0x1}, {0x1ce9580, 0x0, 0x0}, ...}, ...)
        	/go/src/github.com/containerd/nerdctl/pkg/composer/composer.go:86 +0x3bc
        github.com/containerd/nerdctl/v2/pkg/cmd/compose.New(_, {0x0, 0x0, {0x12712ed, 0x1f}, {0x7ffe39189e34, 0xc}, {0x124f988, 0x9}, {0x1261034, ...}, ...}, ...)
        	/go/src/github.com/containerd/nerdctl/pkg/cmd/compose/compose.go:143 +0x465
        main.composeBuildAction(0xc0009f8908, {0x1ce9580, 0x0, 0x0})
        	/go/src/github.com/containerd/nerdctl/cmd/nerdctl/compose_build.go:68 +0x3b9
        github.com/spf13/cobra.(*Command).execute(0xc0009f8908, {0xc0000d0150, 0x0, 0x0})
        	/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0xaca
        github.com/spf13/cobra.(*Command).ExecuteC(0xc0006ecc08)
        	/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
        github.com/spf13/cobra.(*Command).Execute(...)
        	/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
        main.xmain()
        	/go/src/github.com/containerd/nerdctl/cmd/nerdctl/main.go:136 +0x8f
        main.main()
        	/go/src/github.com/containerd/nerdctl/cmd/nerdctl/main.go:119 +0x13
        
--- FAIL: TestComposePushAndPullWithCosignVerify (1.31s)
FAIL cmd/nerdctl.TestComposePushAndPullWithCosignVerify (re-run 1) (1.31s)

@apostasie
Copy link
Copy Markdown
Contributor Author

=== RUN   TestComposePushAndPullWithCosignVerify
    testutil.go:567: buildkitHost="unix:///run/buildkit-nerdctl-test/buildkitd.sock"
=== PAUSE TestComposePushAndPullWithCosignVerify
=== CONT  TestComposePushAndPullWithCosignVerify
    compose_run_linux_test.go:486: projectName="nerdctl-compose-test689402566"
    compose_run_linux_test.go:490: assertion failed: res.ExitCode is not exitCode: panic: runtime error: index out of range [1] with length 1
        
        goroutine 1 [running]:
        github.com/compose-spec/compose-go/utils.GetAsEqualsMap({0xc0008d49c0, 0xd, 0x30?})
        	/go/pkg/mod/github.com/compose-spec/compose-go@v1.20.2/utils/stringutils.go:46 +0xfd
        github.com/compose-spec/compose-go/cli.WithOsEnv(0xc00050c380)
        	/go/pkg/mod/github.com/compose-spec/compose-go@v1.20.2/cli/options.go:231 +0x2c
        github.com/compose-spec/compose-go/cli.NewProjectOptions({0xc0002e00e0, 0x1, 0x1}, {0xc0008d2660, 0x6, 0x10?})
        	/go/pkg/mod/github.com/compose-spec/compose-go@v1.20.2/cli/options.go:93 +0xcf
        github.com/containerd/nerdctl/v2/pkg/composer.New({{0x0, 0x0}, {0x0, 0x0}, {0xc0002e00e0, 0x1, 0x1}, {0x1ce9580, 0x0, 0x0}, ...}, ...)
        	/go/src/github.com/containerd/nerdctl/pkg/composer/composer.go:86 +0x3bc
        github.com/containerd/nerdctl/v2/pkg/cmd/compose.New(_, {0x0, 0x0, {0x12712ed, 0x1f}, {0x7ffe39189e34, 0xc}, {0x124f988, 0x9}, {0x1261034, ...}, ...}, ...)
        	/go/src/github.com/containerd/nerdctl/pkg/cmd/compose/compose.go:143 +0x465
        main.composeBuildAction(0xc0009f8908, {0x1ce9580, 0x0, 0x0})
        	/go/src/github.com/containerd/nerdctl/cmd/nerdctl/compose_build.go:68 +0x3b9
        github.com/spf13/cobra.(*Command).execute(0xc0009f8908, {0xc0000d0150, 0x0, 0x0})
        	/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0xaca
        github.com/spf13/cobra.(*Command).ExecuteC(0xc0006ecc08)
        	/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
        github.com/spf13/cobra.(*Command).Execute(...)
        	/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
        main.xmain()
        	/go/src/github.com/containerd/nerdctl/cmd/nerdctl/main.go:136 +0x8f
        main.main()
        	/go/src/github.com/containerd/nerdctl/cmd/nerdctl/main.go:119 +0x13
        
--- FAIL: TestComposePushAndPullWithCosignVerify (1.31s)
FAIL cmd/nerdctl.TestComposePushAndPullWithCosignVerify (re-run 1) (1.31s)

Yep, this is definitely a positive.
Something about the env in compose - will check ASAP.

Signed-off-by: apostasie <spam_blackhole@farcloser.world>
@apostasie
Copy link
Copy Markdown
Contributor Author

apostasie commented Jun 18, 2024

@AkihiroSuda
The reason it was failing was a stupid typo on my side: https://github.com/containerd/nerdctl/compare/25dc5e5dbadb6308a6fd7b74033a37d05edb387f..6d0c8f872d86f625bfb9bc24e06ef66f4d139b2f

Nevertheless, it seems to me that we did uncover a bug in compose with it.

This below assumes that os.Environ() returns strings that contains "=" - which is clearly not always true:

// WithOsEnv imports environment variables from OS
func WithOsEnv(o *ProjectOptions) error {
	for k, v := range utils.GetAsEqualsMap(os.Environ()) {
		if _, set := o.Environment[k]; set {
			continue
		}
		o.Environment[k] = v
	}
	return nil
}

Then calling GetAsEqualsMap in that case gets the index out of bound and the crash.

https://github.com/compose-spec/compose-go/blob/main/cli/options.go#L238

Feel like reporting it over there? reported over there

Copy link
Copy Markdown
Member

@AkihiroSuda AkihiroSuda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@AkihiroSuda AkihiroSuda merged commit 040c13b into containerd:main Jun 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci e.g., CI failure area/cosign cosign

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tests should not use Setenv as it prevents parallelization

2 participants