integration: basic s3 cache test by crazy-max · Pull Request #3398 · moby/buildkit

crazy-max · 2022-12-13T11:06:01Z

Adds basic integration test for s3 cache so we can get rid of the one from hack folder. Also pin busybox in current azblob_test related to #3469.

cc @bpaquet

crazy-max · 2023-01-08T03:39:21Z

Also pin busybox in current azblob_test related to #3469.

Has been fixed in #3475, removing that commit.

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

bpaquet · 2023-01-09T19:44:45Z

client/client_test.go

 	testBasicCacheImportExport(t, sb, []CacheOptionsEntry{im}, []CacheOptionsEntry{ex})
 }

+func testBasicS3CacheImportExport(t *testing.T, sb integration.Sandbox) {


This is only one case. @crazy-max do you want help for others?

I think the basic one is fine for this use case:

buildkit/client/client_test.go

Lines 4556 to 4616 in 05c4d57

func testBasicCacheImportExport(t *testing.T, sb integration.Sandbox, cacheOptionsEntryImport, cacheOptionsEntryExport []CacheOptionsEntry) {

requiresLinux(t)

c, err := New(sb.Context(), sb.Address())

require.NoError(t, err)

defer c.Close()

busybox := llb.Image("busybox:latest")

st := llb.Scratch()

run := func(cmd string) {

st = busybox.Run(llb.Shlex(cmd), llb.Dir("/wd")).AddMount("/wd", st)

}

run(`sh -c "echo -n foobar > const"`)

run(`sh -c "cat /dev/urandom | head -c 100 | sha256sum > unique"`)

def, err := st.Marshal(sb.Context())

require.NoError(t, err)

destDir := t.TempDir()

_, err = c.Solve(sb.Context(), def, SolveOpt{

Exports: []ExportEntry{

{

Type: ExporterLocal,

OutputDir: destDir,

},

},

CacheExports: cacheOptionsEntryExport,

}, nil)

require.NoError(t, err)

dt, err := os.ReadFile(filepath.Join(destDir, "const"))

require.NoError(t, err)

require.Equal(t, string(dt), "foobar")

dt, err = os.ReadFile(filepath.Join(destDir, "unique"))

require.NoError(t, err)

ensurePruneAll(t, c, sb)

destDir = t.TempDir()

_, err = c.Solve(sb.Context(), def, SolveOpt{

Exports: []ExportEntry{

{

Type: ExporterLocal,

OutputDir: destDir,

}},

CacheImports: cacheOptionsEntryImport,

}, nil)

require.NoError(t, err)

dt2, err := os.ReadFile(filepath.Join(destDir, "const"))

require.NoError(t, err)

require.Equal(t, string(dt2), "foobar")

dt2, err = os.ReadFile(filepath.Join(destDir, "unique"))

require.NoError(t, err)

require.Equal(t, string(dt), string(dt2))

}

If you think about ones specific to s3, we can do that in follow-ups.

crazy-max · 2023-01-30T19:39:44Z

@bpaquet As discussed with @tonistiigi I will open a follow-up to check the bucket content like you've done. If you want to do it, let me know.

guillenotfound · 2023-02-02T22:27:40Z

Is this fixing importing cache from several targets (multi stage builds)? Currently that's not working very well and I cannot find much documentation to ping-pong what's going on.

crazy-max added the status/do-not-merge label Dec 13, 2022

crazy-max force-pushed the test-s3-cache branch from c42d841 to 2b76f8c Compare January 7, 2023 14:46

crazy-max changed the title ~~cache: native s3 cache integration tests~~ integration: basic s3 cache test Jan 7, 2023

crazy-max force-pushed the test-s3-cache branch 2 times, most recently from e4b5c2c to 3782abf Compare January 7, 2023 15:13

crazy-max added area/testing and removed status/do-not-merge labels Jan 7, 2023

crazy-max marked this pull request as ready for review January 7, 2023 16:53

crazy-max force-pushed the test-s3-cache branch 2 times, most recently from 53b4e00 to 0aaf62b Compare January 8, 2023 11:35

crazy-max added 4 commits January 8, 2023 12:36

integration: expose backend logs in sandbox interface

3d93d64

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

integration: add runCmd and randomString utils

c03ccb8

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

integration: basic s3 cache test

35e870d

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

hack: remove s3_test

05c4d57

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

crazy-max force-pushed the test-s3-cache branch from 0aaf62b to 05c4d57 Compare January 8, 2023 11:36

crazy-max requested a review from tonistiigi January 9, 2023 10:10

crazy-max mentioned this pull request Jan 9, 2023

integration: basic azblob cache test #3477

Merged

bpaquet reviewed Jan 9, 2023

View reviewed changes

crazy-max requested a review from bpaquet January 24, 2023 12:16

crazy-max added the needs/follow-up label Jan 30, 2023

tonistiigi approved these changes Jan 30, 2023

View reviewed changes

crazy-max merged commit 9b28f40 into moby:master Jan 30, 2023

crazy-max deleted the test-s3-cache branch January 30, 2023 19:39

crazy-max mentioned this pull request Mar 14, 2023

[0.11 backport] integration: enforce features compat through env vars #3713

Merged

This was referenced Jul 17, 2023

[0.11] vendor: github.com/containerd/containerd v1.6.21 #3884

Merged

[0.11 backport] azblob and s3 integration tests #4030

Merged

crazy-max mentioned this pull request Aug 8, 2024

metrics: add metrics for bake command docker/buildx#2610

Merged

crazy-max mentioned this pull request Apr 14, 2025

feat: Support GCS remotecache #5910

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

integration: basic s3 cache test#3398

integration: basic s3 cache test#3398
crazy-max merged 4 commits intomoby:masterfrom
crazy-max:test-s3-cache

crazy-max commented Dec 13, 2022 •

edited

Loading

Uh oh!

crazy-max commented Jan 8, 2023

Uh oh!

bpaquet Jan 9, 2023

Uh oh!

crazy-max Jan 10, 2023

Uh oh!

crazy-max commented Jan 30, 2023

Uh oh!

guillenotfound commented Feb 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	func testBasicCacheImportExport(t *testing.T, sb integration.Sandbox, cacheOptionsEntryImport, cacheOptionsEntryExport []CacheOptionsEntry) {
	requiresLinux(t)
	c, err := New(sb.Context(), sb.Address())
	require.NoError(t, err)
	defer c.Close()

	busybox := llb.Image("busybox:latest")
	st := llb.Scratch()

	run := func(cmd string) {
	st = busybox.Run(llb.Shlex(cmd), llb.Dir("/wd")).AddMount("/wd", st)
	}

	run(`sh -c "echo -n foobar > const"`)
	run(`sh -c "cat /dev/urandom \| head -c 100 \| sha256sum > unique"`)

	def, err := st.Marshal(sb.Context())
	require.NoError(t, err)

	destDir := t.TempDir()

	_, err = c.Solve(sb.Context(), def, SolveOpt{
	Exports: []ExportEntry{
	{
	Type: ExporterLocal,
	OutputDir: destDir,
	},
	},
	CacheExports: cacheOptionsEntryExport,
	}, nil)
	require.NoError(t, err)

	dt, err := os.ReadFile(filepath.Join(destDir, "const"))
	require.NoError(t, err)
	require.Equal(t, string(dt), "foobar")

	dt, err = os.ReadFile(filepath.Join(destDir, "unique"))
	require.NoError(t, err)

	ensurePruneAll(t, c, sb)

	destDir = t.TempDir()

	_, err = c.Solve(sb.Context(), def, SolveOpt{
	Exports: []ExportEntry{
	{
	Type: ExporterLocal,
	OutputDir: destDir,
	}},
	CacheImports: cacheOptionsEntryImport,
	}, nil)
	require.NoError(t, err)

	dt2, err := os.ReadFile(filepath.Join(destDir, "const"))
	require.NoError(t, err)
	require.Equal(t, string(dt2), "foobar")

	dt2, err = os.ReadFile(filepath.Join(destDir, "unique"))
	require.NoError(t, err)
	require.Equal(t, string(dt), string(dt2))
	}

Conversation

crazy-max commented Dec 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

crazy-max commented Jan 8, 2023

Uh oh!

bpaquet Jan 9, 2023

Choose a reason for hiding this comment

Uh oh!

crazy-max Jan 10, 2023

Choose a reason for hiding this comment

Uh oh!

crazy-max commented Jan 30, 2023

Uh oh!

guillenotfound commented Feb 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

crazy-max commented Dec 13, 2022 •

edited

Loading