Okay, I've found a bug in my proposed implementation, when the daemon is restarted, docker may try to use persisted ports in other containers, preventing container with persisted port to start correctly. This is also an issue if one calls run with -p [49153-65535]:[something].

This problem could be addressed by allocating ports for containers with explicit host ports in HostConfig first when starting the daemon. I'll try to add this to this PR.

I updated the PR including a test case and a fix to the issue I described in my previous comment. The solution involves initializing the network of containers with explicit host ports before containers with dynamically allocated ports when starting the daemon.
Any idea if there's a possibility of this proposal being accepted?

Docs LGTM - @fredlf @jamtur01

just one thing, I think it would be good to mention that container restart means only docker restart, or the container/(id)/restart API call .

I'm also not sure if that includes rebooting the computer, or restarting the Docker daemon.

Docs LGTM

Seems like instead of a flag we should just keep the port for the life of the container no matter what.

@cpuguy83 If that's the way to go I can change this pull request (or create another one) removing the flag and making this the default behavior. Just let me know.

I'll defer to the core maintainers on this, and @aluzzardi who was working on the "stable networking" stuff, which I think this would fall under.

cc @jfrazelle @crosbymichael @icecrime

First of all, thank you for the contribution: it's a sensible feature being implemented in a sensible way!

Now, without an extensive review: if I understand correctly, what the implementation does is take the -p X specification, elect the host port Y to map to, and persist that just as if I originally launched the container with -p Y:X, is that right?

That makes we wonder on two things (which may very well end up being dismissed as irrelevant):

• Is this gonna make us "lie" in the inspect output by pretending ports were bounds when they were really just exposed? I know some tools (swarm, maybe compose) are relying on this output.
• Is that to say that if I run my container with --persist-ports and that the elected port is no longer available upon restart, the container fails to start (like it would with an actual -p X:Y)? That would make sense I guess, but definitely go against making host port persistency the default.

You're right, after the container starts it's as if it was started with -p Y:X Being Y the port chosen by the dynamic allocator the first time the container started.

The inspect output is slightly different. NetworkSettings is the same. HostConfig though shows as:
With -p 9999:80:

"HostConfig": {
    "PersistPorts": false,
    "PortBindings": {
        "80/tcp": [
            {
                "HostIp": "",
                "HostPort": "9999"
            }
        ]
    }
}

With --persist-ports -p 80:

"HostConfig": {
    "PersistPorts": true,
    "PortBindings": {
        "80/tcp": [
            {
                "HostIp": "0.0.0.0",
                "HostPort": "49161"
            }
        ]
    }
}

But I don't see that exactly as a lie. It's true that the host port is now part of the configuration as it will remain the same.
Having HostIp set wasn't intentional. It was a side effect of setting the whole PortBindings entry, I can change the code to remove it before saving, but I don't think having it set is a problem.

About your last question, yes, the container would fail to start the same way it would if it was started with -p Y:X. And I agree that this would be unexpected if this was the default behavior. I think the proposed flag is be the safest way to introduce this change and maybe consider changing the default in the future.

That said, some of the changes in this pull request regards preventing docker itself from using a port that was persisted, specially when the daemon starts and multiple containers have to be restarted. This prevents "port already allocated" errors from showing up unless some external process is using it.

Indeed, Swarm should be aware of this change since it uses that information to know which ports are available in the cluster.

Also, stable networking was dropped back then since it introduced a UX regression.

I think that if we do this, we should also persist the IPs since it would be weird to keep the same ports but change the IP.

@aluzzardi Is this the code in swarm in question: https://github.com/docker/swarm/blob/master/scheduler/filter/port.go#L34?
If so, I don't think introducing this flag will break anything.

Containers started with --persist-ports will have their HostConfig.PortBindings correctly with HostIp and HostPort set by docker the first time they were started. This way if the same host port is explicitly exposed by another container trying to be scheduled, portAlreadyInUse will correctly return true and another node will be chosen.

However, just from reading the code (I haven't tried it), I can see that there's a possible bug that may already happen today.
Suppose I have a swarm with two nodes A and B both without any containers yet. Then I run the following commands pointing to the swarm:
$ docker run -d -p :80 busybox top (container 1)
Container 1 will end up in either A or B, and as it's using dynamic port allocation it will have allocated port 49153.
Now I run:
$ docker run -d -p 49153:80 busybox top (container 2)
Container 2 also may end up in either A or B, as portAlreadyInUse will return false for both nodes A and B.
If the chosen node happens to be the same used to start the container 1, container 2 will fail to start. This will happen because Docker will only show dynamically allocated host ports in NetworkSettings.Ports and not in HostConfig.PortBindings, the latter being the only one verified by portAlreadyInUse.
As I said, this is unrelated to this pull request. Actually, using --persist-ports would prevent this bug from happening.

@cezarsa Thank you very much for looking into it!

Would you mind opening an issue on Swarm regarding the bug you found?

Docs LGTM. You can take or leave my suggestion.

Has merge conflicts.

I wonder whether this should be the default or not, that way no need for another flag. /cc @crosbymichael @aluzzardi

I'll reiterate what I suggest:

I think we should not add a new flag, and simply allocate ports on create, and free ports on removal by default. That way ports will be persistent. The only bad thing is that if you have a huge ton of stopped containers, then you have less available ports, but it should be fine in practice.

@tiborvass I agree with you on that (attaching ports lifecycle to create/remove). It might break some existing feature and/or design principle, though.

Featuring ports persistence would allow me to get rid of same logic in docker-fw; creation should indeed fail if those persisted ports are not available.

@tiborvass It seems a little odd to reserve a port that is not actually in use by a running container to be considered "allocated". Per @icecrime's point, it seems problematic that every container launched on a daemon with --persist-ports effectively becomes a container started with -p X:Y.

The main problem here is that allocating something with an "ephemeral port" says two things:

1. Use any port, I don't care.
2. Only error when binding the port if there are no available ports.

Locking down this seems to make these no longer ephemeral port allocations, since it breaks both of the above invariants.

I do understand the benefit here (of eliminating SPOF for bind notification), but couldn't that be done by managing the port allocation externally and then issue container creation/restart commands with -p X:Y?

@stevvooe Interesting point you're making there :) Made me analyze a little more why I was thinking keeping the port on stop made sense.

You're saying that the ports are supposed to be ephemeral. I agree. They would still be, at each container creation. Which means the creator of the container can never know what port it will get, and that's good.

The container's fs is ephemeral as well, but it's gone only on removal, not on stop. The argument for keeping the port in memory for stopped containers, is that a stopped container is still the same container, and likely to be restarted.

I guess the main question is: should ports "live" and "die" on their own as they are used (like, on another scale, containers), or are ports part of the state of a container?

Beyond the philosophical debate, what's the use-case for not having the same port after a restart? Or, what would be the usecase that we would kill, by persisting the ports?

@tiborvass Your analysis of my response is accurate.

Or, what would be the usecase that we would kill, by persisting the ports?

This is a key question. IMHO, we kill the use case of ephemeral ports.

The main issue here is the behavior upon restart of a container. Does docker take measures to ensure that the ephemeral port is always ready for use? Or, does it re-allocate a new port on a collision, guaranteeing the ports are actually ephemeral? If we choose the latter, it makes keeping the port mapping around an optimization to avoid sending port updates to consuming services.

Two things follow from this:

1. The port mapping should not be stored in the same way as configs from -p X:Y.
2. Whatever does manage the port mapping should be able to mark that mapping as ephemeral and reassign it, if needed.

I am not sure if this proposal follows these two items, but they should be considered.

Collective review with @LK4D4 @calavera @jfrazelle @crosbymichael @tiborvass

The debate around this PR is that you'll always be left having to manage something yourself:

• Either port allocation (when you don't use --persist-ports)
• Either the fallback strategy at start if the port is already allocated (when you use --persist-ports)

If we proceed with this, stopped containers will eventually fill the automatic port allocation pool and make it impossible to run any new containers. We can't really force people to garbage collect their containers (although it's good practice), so this comes with new risks too.

Sorry, but we're gonna close this. Thanks again for the contribution though!

Docker already handles automatic port allocation, so why should an external system have to do the same thing unnecessarily.

I think there could be a middle ground if the user provides a port range from which an automatic selection can be made to persist (and a recommendation that it be outside of the default ephemeral range).

e.g.
# docker run -p 8000-10000:80 <image>
or
# docker run -P --portrange 8000-10000 <image>

If this custom range is provided, the container would be started as if with -p X:Y and would persist on restarts...

Thoughts?