Skip to content

Network: add support for 'dangling' filter#31551

Merged
vdemeester merged 1 commit intomoby:masterfrom
KarthikNayak:dry_run
Feb 28, 2019
Merged

Network: add support for 'dangling' filter#31551
vdemeester merged 1 commit intomoby:masterfrom
KarthikNayak:dry_run

Conversation

@KarthikNayak
Copy link
Copy Markdown
Contributor

Closes #30825

Like its counterpart in images and volumes, introduce the dangling
filter while listing networks. When the filter value is set to true,
only networks which aren't attached to containers and aren't builtin
networks are shown. When set to false, all builtin networks and
networks which are attached to containers are shown.

Signed-off-by: Karthik Nayak Karthik.188@gmail.com


- How I did it

Mostly by mimicking the existing code, and writing new code for supporting this feature.

- How to verify it

Either by running the unit test included, or by simply building and running
docker network ls --filter dangling=true and docker network ls --filter dangling=false

- Description for the changelog

Network: add support for 'dangling' filter

- A picture of a cute animal (not mandatory but encouraged)

cat
Credits: Flickr/NEKOFighter

Comment thread api/server/router/network/filter.go Outdated
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure whether this part is needed.
danglingOnly is bool, which means it can be true or false.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, will remove that 👍

Comment thread api/server/router/network/filter.go Outdated
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE: This is a suggestion.
I would prefer get rid of the switch block and do the following:

for _, nw := range nws {
    if !runconfig.IsPreDefinedNetwork(nw.Name) && len(nw.Containers) == 0 && danglingOnly {
        retNws = append(retNws, nw)
    }
}

I think it'll work.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But how will this work when dangling=false. In that case we'd have to append non-dangling networks to retNws. Which is done only when, either, the network has been attached to more than a container or is a pre-defined network.

Copy link
Copy Markdown
Contributor

@boaz0 boaz0 Mar 6, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I knew I missed something.
What about setting a filter function instead of using a condition.

filterFunc := func(nw types.NetworkResource) bool {
    if danglingOnly {
        return !runconfig.IsPreDefinedNetwork(nw.Name) && len(nw.Containers) == 0
    }
    return runconfig.IsPreDefinedNetwork(nw.Name) || len(nw.Containers) > 0
}
...
for _, nw := range nws {
    if filterFunc(nw) {
        retNws = append(retNws, nw)
    }
}

Even though I believe it's possible to put those conditions in one line, I am afraid it won't be readable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That works too, but I don't see why we need to do this?

I mean the code currently tries to be consistent with the existing function filterNetworksByType, and staying consistent would mean better readability. Not that I'm against your suggestion, I just don't see any advantage in doing so.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i like @ripcurld0 idea 😊

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vdemeester I'm ok with changing to that, but that wouldn't be consistent with the existing code, that's all.

@AkihiroSuda AkihiroSuda added status/1-design-review status/failing-ci Indicates that the PR in its current state fails the test suite and removed status/0-triage labels Mar 6, 2017
Comment thread api/server/router/network/filter.go Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not work for swarm networks, because nw.Containers does not contain containers running on other nodes

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, any suggestions how to go about this?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would mean dropping this commit and re-working, i.e. adding the DryRun option to RemoveNetworkRequest and perhaps then have a this filter just call that?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, unfortunately..

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, ListTasksRequest does not work for docker run containers.
i.e.

$ docker network create --driver overlay --attachable foo
$ docker run --network foo ...

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect it to work for docker run containers attached to networks. Those are represented by tasks, and I don't know of anything that filters them out in swarmkit (maybe there is something filtering them in the engine?).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So basically this PR would be dependent on #24186, right?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably yes. But it would be OK if ListTasksRequest work.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that #31710 is accepted, I'll finish this soon 👍

@allencloud
Copy link
Copy Markdown
Contributor

I think this PR changes filters for GET /networks API in the daemon side, so we should change swagger.yml.
In addition, docker_network_ls.md should be modified as well. @KarthikNayak

@KarthikNayak
Copy link
Copy Markdown
Contributor Author

@AkihiroSuda @aaronlehmann Do have a second look.

Comment thread api/server/router/network/filter.go Outdated
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe these fields are provided by the gossip layer and are eventually consistent. Is it a problem if the dangling filter may provide results that are slightly out of date?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I know enough to answer that. I'll look around.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it should be a problem as long as the consumer knows this is potentially racey.
Trying to rm a network from this list wouldn't really pose a problem except for potentially racey behavior from the state propagation, but this problem would exist in any case.

@AkihiroSuda
Copy link
Copy Markdown
Member

How does it work with docker run containers on other nodes?
#31710 does not seem to support them

@AkihiroSuda AkihiroSuda removed the status/failing-ci Indicates that the PR in its current state fails the test suite label Mar 23, 2017
@KarthikNayak
Copy link
Copy Markdown
Contributor Author

@AkihiroSuda It won't, since this is based off that patch

@AkihiroSuda
Copy link
Copy Markdown
Member

Then please add a doc so as to clarify that dangling network is not guaranteed to be dangling?

@KarthikNayak
Copy link
Copy Markdown
Contributor Author

I could do that, would that be ok though? @AkihiroSuda

@AkihiroSuda
Copy link
Copy Markdown
Member

@KarthikNayak

I think it is ok

cc @sanimej @aaronlehmann @thaJeztah

@thaJeztah
Copy link
Copy Markdown
Member

Yes, the situation with (e.g.) volumes is no different; a volume can be "dangling" when filtering, but there can be a race where it's being used by a container directly after that.

Moving this to code review

@thaJeztah
Copy link
Copy Markdown
Member

looks like it needs a rebase @KarthikNayak 😢

@KarthikNayak
Copy link
Copy Markdown
Contributor Author

Already on it :)

Comment thread api/swagger.yaml Outdated
Copy link
Copy Markdown
Contributor

@allencloud allencloud Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add "0" and "1" ? Print->Prints>
WDYT

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do s/Print/Prints.

About '0' and '1', I'm a little confused.

~ docker image ls --filter dangling=0
Error response from daemon: Invalid filter 'dangling=[0]'

~ docker volume ls --filter dangling=0
DRIVER              VOLUME NAME

So I'm on the fence with that

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I am just referenced to https://github.com/docker/docker/blob/master/api/swagger.yaml#L5089-L5091 for image prune dangling:

- `dangling=<boolean>` When set to `true` (or `1`), prune only
               unused *and* untagged images. When set to `false`
               (or `0`), all unused images are pruned.

@vdemeester
Copy link
Copy Markdown
Member

ping @mavenugo @fcrisciani @thaJeztah @vieux what is the status with this one ? 👼

@vieux
Copy link
Copy Markdown
Contributor

vieux commented Aug 23, 2017

@vdemeester I'm still 👍 for this feature

@thaJeztah
Copy link
Copy Markdown
Member

I'm still 👍 as well, but it needs a rebase and the docs, completion-scripts need to be moved to https://github.com/docker/cli @KarthikNayak

@cpuguy83
Copy link
Copy Markdown
Member

I went ahead and rebased this, updated based on code comments.

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Left some comments

Comment thread api/server/router/network/filter.go Outdated
Comment thread api/server/router/network/filter.go Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonder if we should be explicit here, and add danglingOnly = false. Perhaps because I'm used to "fall-through" being the default in other languages, but I had to look twice to interpret this correctly 😊

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's already false?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I know, it's probably just me, reading;

switch {
  case foo:
    dofoo()
  case bar:
    dobar()
  case baz:
  default:
    dosomethingelse()
}

I always tend to first read that as baz: falls through to default: (so error in this case)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually; should this use filter.ExactMatch() here?

moby/daemon/prune.go

Lines 185 to 191 in 2a7388a

if pruneFilters.Contains("dangling") {
if pruneFilters.ExactMatch("dangling", "false") || pruneFilters.ExactMatch("dangling", "0") {
danglingOnly = false
} else if !pruneFilters.ExactMatch("dangling", "true") && !pruneFilters.ExactMatch("dangling", "1") {
return nil, invalidFilter{"dangling", pruneFilters.Get("dangling")}
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why one would be better than the other. This filters package is pretty confusing.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include() is deprecated; can you change to Contains()?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it a concern that when filtering for "dangling", the returned results will be different than without? (i.e. it having "detailed" information)?

Looking at buildDetailedNetworkResources(), and how we use that information during filtering (we're only interested in knowing if containers or services are attached);

func (n *networkRouter) buildDetailedNetworkResources(nw libnetwork.Network, verbose bool) *types.NetworkResource {
if nw == nil {
return &types.NetworkResource{}
}
r := n.buildNetworkResource(nw)
epl := nw.Endpoints()
for _, e := range epl {
ei := e.Info()
if ei == nil {
continue
}
sb := ei.Sandbox()
tmpID := e.ID()
key := "ep-" + tmpID
if sb != nil {
key = sb.ContainerID()
}
r.Containers[key] = buildEndpointResource(tmpID, e.Name(), ei)
}
if !verbose {
return r
}
services := nw.Info().Services()
r.Services = make(map[string]network.ServiceInfo)
for name, service := range services {
tasks := []network.Task{}
for _, t := range service.Tasks {
tasks = append(tasks, network.Task{
Name: t.Name,
EndpointID: t.EndpointID,
EndpointIP: t.EndpointIP,
Info: t.Info,
})
}
r.Services[name] = network.ServiceInfo{
VIP: service.VIP,
Ports: service.Ports,
Tasks: tasks,
LocalLBIndex: service.LocalLBIndex,
}
}
return r
}

Wondering if we should have a simplified version, or at least reset the Containers and Services information afterwards, so that the information that's returned for the network is consistent.

@cpuguy83
Copy link
Copy Markdown
Member

This network code is awful :( We should not be dealing with libnetwork types in the API layer.

@olljanat
Copy link
Copy Markdown
Contributor

olljanat commented Jan 9, 2019

@KarthikNayak looks like flaky test.
Created item #38521 but I will also do some stress tests for this PR.

@codecov
Copy link
Copy Markdown

codecov bot commented Jan 9, 2019

Codecov Report

❗ No coverage uploaded for pull request base (master@de7172b). Click here to learn what that means.
The diff coverage is 80.95%.

@@            Coverage Diff            @@
##             master   #31551   +/-   ##
=========================================
  Coverage          ?   36.45%           
=========================================
  Files             ?      613           
  Lines             ?    45873           
  Branches          ?        0           
=========================================
  Hits              ?    16722           
  Misses            ?    26856           
  Partials          ?     2295

@olljanat
Copy link
Copy Markdown
Contributor

I found couple of other PRs too where TestStartReturnCorrectExitCode has failed and also did try to run it 100 times on loop which this PR passed just fine.

@thaJeztah @cpuguy83 can we get this one merged before its second anniversary? 😮

Comment thread daemon/network/filter_test.go Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry to do this, but do you think we can alter the test to actually test for which networks should be returned? Can you also add a network that is not a built-in but is not dangling?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cpuguy83 Haha, I'll get to it soon 👍

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cpuguy83 Done! 👍

Copy link
Copy Markdown
Member

@cpuguy83 cpuguy83 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah
Copy link
Copy Markdown
Member

whoops; looks like this needs gofmt;

12:25:41 daemon/network/filter_test.go:1::warning: file is not gofmted with -s (gofmt)
12:25:42 Build step 'Execute shell' marked build as failure

@KarthikNayak can you run gofmt on that file to make CI pass?

Like its counterpart in images and volumes, introduce the dangling
filter while listing networks. When the filter value is set to true,
only networks which aren't attached to containers and aren't builtin
networks are shown. When set to false, all builtin networks and
networks which are attached to containers are shown.

Signed-off-by: Karthik Nayak <Karthik.188@gmail.com>
@KarthikNayak
Copy link
Copy Markdown
Contributor Author

whoops; looks like this needs gofmt;

12:25:41 daemon/network/filter_test.go:1::warning: file is not gofmted with -s (gofmt)
12:25:42 Build step 'Execute shell' marked build as failure

@KarthikNayak can you run gofmt on that file to make CI pass?

Definitely! Fixed and pushed :)

@albers
Copy link
Copy Markdown
Member

albers commented Mar 9, 2019

Thanks for implementing this feature @KarthikNayak.

@KarthikNayak
Copy link
Copy Markdown
Contributor Author

@albers thanks for finishing it up with the bash completion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.