Skip to content

Conversation

@rvolosatovs
Copy link
Contributor

@rvolosatovs rvolosatovs commented Jun 23, 2021

- What I did
Add type parameter to /system/df

- How I did it
Internally allow more fine-grained control over choice of objects to compute disk usage from, add types URL parameter.

- How to verify it

$ docker run alpine # or create a container in any other way you prefer
$ curl -s --unix-socket /var/run/docker.sock 'http://localhost/system/df?types=image&type=container' | jq # should see image and container data
$ curl -s --unix-socket /var/run/docker.sock 'http://localhost/system/df?type=container&type=invalid' | jq # should see an error:
{
  "message": "unknown object type: invalid"
}
$ curl -s --unix-socket /var/run/docker.sock 'http://localhost/system/df?type=container' | jq # should see only container data
$ curl -s --unix-socket /var/run/docker.sock 'http://localhost/system/df' | jq # should see data, just like before

- Description for the changelog

/system/df endpoint now supports type URL parameter, which gives control over object types to compute disk usage of

@rvolosatovs rvolosatovs force-pushed the system_df_types branch 6 times, most recently from 623bed8 to e620e9b Compare June 28, 2021 10:24
@rvolosatovs rvolosatovs marked this pull request as ready for review June 28, 2021 10:26
@rvolosatovs rvolosatovs requested a review from thaJeztah June 28, 2021 10:26
@rvolosatovs
Copy link
Contributor Author

Blocking #42560

Copy link
Contributor

@fredericdalleau fredericdalleau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rvolosatovs rvolosatovs force-pushed the system_df_types branch 2 times, most recently from 87423ba to 61e98a4 Compare July 9, 2021 17:30
@thaJeztah
Copy link
Member

Arf.. I should've looked at Jenkins looks like the test is failing 😞

@thaJeztah thaJeztah dismissed their stale review July 26, 2021 16:05

test failing

@rvolosatovs
Copy link
Contributor Author

rvolosatovs commented Jul 26, 2021

Sorry, should have checked CI!

@rvolosatovs rvolosatovs marked this pull request as draft July 26, 2021 16:52
@rvolosatovs rvolosatovs force-pushed the system_df_types branch 2 times, most recently from 917946d to 88ab006 Compare July 26, 2021 20:20
@rvolosatovs
Copy link
Contributor Author

rvolosatovs commented Jul 26, 2021

It seems that there's no way to use the testing approach I chose, since the store is modified in parallel to the test execution unless we use -p=1 like in #42594, which I would rather not do. Is there a simple way to setup the daemon with a custom store for a particular test? Or perhaps some way to acquire a global lock for the duration of the test?

@cpuguy83
Copy link
Member

You can setup a separate daemon for the test:

d := daemon.New(t, "--iptables=false")
defer d.Cleanup(t)
d.Start(t) // or d.StartWithBusybox if needed.
defer d.Stop(t)

@rvolosatovs rvolosatovs marked this pull request as ready for review July 26, 2021 22:06
@rvolosatovs rvolosatovs requested a review from thaJeztah July 26, 2021 22:06
@rvolosatovs
Copy link
Contributor Author

You can setup a separate daemon for the test:

d := daemon.New(t, "--iptables=false")
defer d.Cleanup(t)
d.Start(t) // or d.StartWithBusybox if needed.
defer d.Stop(t)

Perfect, I followed this approach!

@rvolosatovs
Copy link
Contributor Author

Looks like 2 test instances still failed 🤔 . This time it seems to be limited to Windows:

disk_usage_test.go:16: [d9303c6a11edc] failed to start daemon with arguments [--data-root D:\gopath\src\github.com\docker\docker\bundles\tmp\TestDiskUsage\d9303c6a11edc\root --exec-root C:\Windows\TEMP\dxr\d9303c6a11edc --pidfile D:\gopath\src\github.com\docker\docker\bundles\tmp\TestDiskUsage\d9303c6a11edc\docker.pid --userland-proxy=true --containerd-namespace d9303c6a11edc --containerd-plugins-namespace d9303c6a11edcp --containerd /var/run/docker/containerd/containerd.sock --host unix://C:\Windows\TEMP\docker-integration\d9303c6a11edc.sock --debug --storage-driver overlay2 --iptables=false] : protocol not available

That is on d.Start(t, "--iptables=false"). Perhaps --iptables=false is not supported on Windows and that's why it fails? Should we check the OS before adding the flag? Is there an alternative one for Windows?

@rvolosatovs rvolosatovs force-pushed the system_df_types branch 2 times, most recently from 2d0aa35 to 61b3e7e Compare July 27, 2021 09:08
@rvolosatovs
Copy link
Contributor Author

Removed --iptables=false flag, since many other tests also don't use it. Added t.Parallel() as well, since we now use a separate daemon.

@rvolosatovs
Copy link
Contributor Author

Looks like iptables was not the problem:

disk_usage_test.go:18: [d81ab14b1027e] failed to start daemon with arguments [--data-root D:\gopath\src\github.com\docker\docker\bundles\tmp\TestDiskUsage\d81ab14b1027e\root --exec-root C:\Windows\TEMP\dxr\d81ab14b1027e --pidfile D:\gopath\src\github.com\docker\docker\bundles\tmp\TestDiskUsage\d81ab14b1027e\docker.pid --userland-proxy=true --containerd-namespace d81ab14b1027e --containerd-plugins-namespace d81ab14b1027ep --containerd /var/run/docker/containerd/containerd.sock --host unix://C:\Windows\TEMP\docker-integration\d81ab14b1027e.sock --debug --storage-driver overlay2] : protocol not available

It looks like tests that do d.Start are actually skipped on Windows, so I followed the same approach.

Let clients choose object types to compute disk usage of.

Signed-off-by: Roman Volosatovs <roman.volosatovs@docker.com>
Co-authored-by: Sebastiaan van Stijn <github@gone.nl>
@rvolosatovs
Copy link
Contributor Author

@cpuguy83 @thaJeztah PTAL

@justincormack justincormack dismissed cpuguy83’s stale review August 2, 2021 13:15

tests now implemented

Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah thaJeztah added this to the 21.xx milestone Aug 2, 2021
@thaJeztah thaJeztah merged commit 656a5e2 into moby:master Aug 2, 2021
@rvolosatovs rvolosatovs deleted the system_df_types branch August 2, 2021 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants