FWIW the single test that failed passed on my local run inside the dev container:

DockerSuite.TestPushBusyboxImage                            1.15
DockerSuite.TestPushEmptyLayer                              0.30
DockerSuite.TestPushInterrupt                               1.07
DockerSuite.TestPushMultipleTags                            1.35   <=========
DockerSuite.TestPushUnprefixedRepo                          0.07
DockerSuite.TestPushUntagged                                0.20
DockerSuite.TestRemoveImageByDigest                         2.18

I'm curious what the use case for this is. I can't think of any reason why you would want docker to use one dynamic range, and non-docker to use another dynamic range.

I don't think this makes sense as a container configuration.
I can use a use for this as a daemon config.

Actually docker still uses both ranges - the primary use case is for production deployments where the specific port used isn't important, but the port should persist over daemon/host/container restarts because the container is behind a reverse proxy (F5 BIG-IP in my case). Using the "static range" provides this persistence of the port after a restart. Dev and Test containers in our environment would use dynamically allocated, ephemeral ports.

The alternative is having some additional 3rd party or custom service that selects and manages ports on all hosts and is injected into the deployment process to select a port on each host at container run time, then run the containers with an explicit port. This is not ideal as Docker already provides a port-selection mechanism.

If you need a static port, why not use a specific port in the -p option (eg -p 1234:80). This is what it's for.

Because then I have to track and manage ports that are in use on every host/service that I deploy, which is something the docker daemon is already doing on my hosts. I don't need the port to be specific, just static. It also doesn't need to be the same port on each host.

@bru7us @icecrime covered why the other PR was closed in #10052 (comment). How does this PR address those points?

I could see something where one specifies a port range then upon restart, it grabs a new port from that range. However, having this be sticky after assignment seems problematic.

@stevvooe , my understanding is that the biggest concern with the other PR seemed to be that we were locking up ports from the ephemeral pool (with stopped containers eventually leading to a port exhaustion situation). My proposal addresses that by implicitly leaving the ephemeral range alone and blends the existing features that the daemon provides to:

• find an available port
• attempt to persist it (i.e. mimic the -p X:Y behavior)
• not keep any port "in use" on a stopped container

It also adds the ability to define any port range for a specific container (or group of containers) from which to search and allocate free ports (outside of the standard/system ephemeral range if that's what you need) without forcing you to track and manage those ports via some other means.

I feel there's a good balance here because if the container is stopped and another container ends up running with the same port while it's down, the first container would simply fail to restart because the port is in use (exactly the same as current [and expected] behavior when a user explicitly starts with -p X:Y.

Ultimately, a user would most likely want to have an 'always' restart-policy when using this feature on a container whose port should remain static. Based on numbered point 2 in your last comment on #10052, this solution only every allocates a port once - using this feature doesn't

TL;DR: You're not forced to manage port allocation externally, nor to manage stopped containers from locking up ports. This is not "port persistence", rather "one-time port selection from a range at create time".

Note the main concern was not about exhaustion of "the" ephemeral port range but rather any ephemeral port range.

I feel there's a good balance here because if the container is stopped and another container ends up running with the same port while it's down, the first container would simply fail to restart because the port is in use (exactly the same as current [and expected] behavior when a user explicitly starts with -p X:Y.

This is the main issue with the proposal. How would one mitigate the failure to restart? Effectively, there is a failure condition on the second start that did not exist in the first start. It seems like it would make more sense to reallocate from the original provided range rather than fail because the container can't rebind the random port it got before.

Otherwise, if you need static ports, use static ports.

I hear what you are saying - but I don't think this change introduces any new failure conditions. -p X:Y currently has the same issue. The problem with simply using static ports is that one then has to actively manage which ports are used and allocated.

The use case for this type of interaction is where you really don't care which port is in use (container will be added to pool config on proxy after startup), but once you have the port, it should be sticky.

Do you think the proposal would seem more holistic if we re-allocate from the range if the originally selected port is in use on restart? That shouldn't be too significant of a change.

OK, latest commit implements your suggestion. I'd really prefer to be able to dump a warning back to the user/client, but passing any non-nil error looks like it would release the port and bail out... so for now we log a warning if we fall back to a re-allocation situation.

I think the documentation update makes it pretty clear on what the expected behavior is when you use the option. We first try to persist the same port, but if it's in use AND we're inside of the custom range for this container, try to allocate a new one.

Alright - after looking at #10052, #12622 (this PR) and #12927, I believe I have conceptualized the solution we're all looking for, which is an effective merge of all 3 proposals and a couple of other tweaks.

I haven't started changing code for this yet, but there shouldn't be much more to change other than what is already covered by these 3 PRs. I'd love to get some community feedback on the concept first:

A user should be able to provide:

• a per-SPEC port range for dynamic allocation (Proposal: Change --publish=SPEC to allow binding to custom host port ranges #12927)
• a per-container port range for dynamic allocation (Proposal: Port Allocations from a Host Range and Host Port Persistence Policies #12622); or
• no range, default ephemeral ports used

No port will ever be 'consumed' by a stopped container.

There should then be a port persistence policy on each container.
e.g. --persist-ports=[ specific | attempt | always ]:
specific is the default and current behavior, where specifically assigned ports (from -p X:Y) are bound to the container config, and a restart will fail if the port is busy.
attempt would apply to all dynamically allocated ports - on restart, we first try to get the port that was previously allocated - if this fails, allocate again from the range that was used for this port. Static ports could still cause a restart failure if busy.
always will persist all dynamically allocated ports in the config as if they were allocated via -p X:Y. Any busy port would cause a failure to restart.

I really think this covers all of the bases and should also alleviate the concerns that were previously raised. Do you have any thoughts on the matter @stevvooe @icecrime @cezarsa @dkjer ?

@bru7us You have appropriately the described the various behaviors and I do like the way you've mapped them. We may want to work on the terminology "attempt" vs "always", as we could swap the terms and definitions and get the same result. Something like a "soft"-bind vs a "hard"-bind.

Also, seems like these options should only apply to dynamic allocations. For example, providing -p X:Y on the command line with attempt is non-sensical. It one provides -p X:P with --persist-ports, it should be an error.

Also, seems like these options should only apply to dynamic allocations. For example, providing -p X:Y on the command line with attempt is non-sensical. It one provides -p X:P with --persist-ports, it should be an error.

Yes, this would only ever apply to dynamic ports, however I think you should still be able to use persistence and static allocations together without error, this should be completely valid:
docker run --persist-ports=[something] --port-range=5000-6000 -p 80:80 -p 8000 -p 8001 -p 8000-9000:8080 ...

@bru7us Why not just keep it simple, and use --persist-ports (no =[something])? The default behavior for port ranges could be changed to what you call 'attempt'. When --persist-ports is specified, the only difference for port ranges would be that a busy previously allocated port would cause a failure.

@dkjer There was some earlier discussion and it seemed to be a non-favorable option to have there be an error condition on second start if the port was busy. I figured having a very explicit policy to apply on first start mitigated that somewhat. I'd be happy enough to push forward with the simple flag - if the maintainers buy off.

@bru7us

Changing the default port-range behavior to 'attempt' shouldn't cause addition error conditions, if it falls back to the original port range, as you described.

Falling back to the original port range is already the default behavior, in #12927 .

Right, but changing the default could be breaking for some other use cases. The current default behavior for dynamic allocation is to always re-allocate.

@stevvooe I've just finished implementation as described above and squashed my commits. I think the updated terminology should be a bit clearer, I went with:
--port-persistence=
'static': Only static ports persist. Dynamic ports will always be re-allocated on restart (current and default)
'soft': Try to preserve the dynamic host ports on restart, re-allocate from original range if port is in use
'hard': Try to preserve the dynamic host ports on restart, error if port is in use

Do you think it might be ready for code/doc-review?

Looks like this code is going to be dead soon due to #13060.
Is it worth my effort to implement this again after libnetwork merges up?

@bru7us It's more critical that we get the design correct first.

Rather than have a flag that applies to all of the arguments for the particular persistence type, why not have the argument dictate the behavior. For example, --persist-ports=<range> would grab a port and re-bind on restart, with an error if its already taken. Something like --port-range=<range> would take the ephemeral approach of providing a guaranteed bind in that range, barring exhaustion. This gets rid of the modes and makes their application per port specification explicit.

@stevvooe Agreed - getting the design pinned down and accepted is indeed critical.

On your last comment - I have a few questions:
Does that stop us from being able to have a per-SPEC range, or do we just have a default bind behavior for those as well?
What happens if the two flags are used together and with overlapping ranges?
Do we lose what we called "soft" bind by only having these two options?

Does that stop us from being able to have a per-SPEC range, or do we just have a default bind behavior for those as well?

I'm not sure what you mean here. Could you clarify?

What happens if the two flags are used together and with overlapping ranges?

We should not allow overlapping port ranges on the command line.

Do we lose what we called "soft" bind by only having these two options?

No. This would be the default behavior for port ranges (this is what we do with -P today and what I suggested above as --port-range).

Does that stop us from being able to have a per-SPEC range, or do we just have a default bind behavior for those as well?
I'm not sure what you mean here. Could you clarify?

So the way I understand your suggestion is this: both --port-range=[range] and --persist-ports=[range] would apply to all dynamic bindings on the container (i.e. where no host port is specified). My latest implementation also provided the ability to specify a host range as -p 8000-9000:80, to which the containers persistence policy would also apply. This range specification is what I mean by 'per-SPEC'.

We should not allow overlapping port ranges on the command line.

When using the argument to define behavior, that makes sense. But if I provided 2 different ranges, how do we decide which range to use for each binding where the host port is not specified?

That is, what happens to container port 90 here?:
--port-range=5000-5999 --persist-ports=6000-6100 -p 80:80 -p 90

With my last implementation, overlaps actually don't matter because we are still using a single global port-map and a single global persistence policy. You can also use both a specified range AND the ephemeral range if you choose to do so (by specifying host port 0 for ephemeral).

No. This would the default behavior for port ranges (this is what we do with -P today and what I suggested above as --port-range).

What we do today is equivalent to the "static" bind, which always allocates a new port on every start (for dynamic ports); "soft" tries to use the previous port if previous allocation was dynamic and only re-allocates a new one if the old is busy.

@bru7us Thanks for your contribution, I'm very sorry that this got invalidated by the recent introduction of docker/libnetwork. I suggest you take the discussion on that repo (mentioning the history here).

Ping @mavenugo @mrjana @aboch: please give this a look, and eventually give guidance to @bru7us if you feel this is an appreciable addition to libnetwork.