cephadm: Set ms bind ipv6 when mon-ip is ipv6 by matthewoliver · Pull Request #35633 · ceph/ceph

matthewoliver · 2020-06-18T01:57:51Z

If you use cephadm bootstrap with an ipv6 mon ip then currently you'll
get into a address family split-brain state, where the mon's messenger
connects and binds to ipv6 but the mgr's binds to ipv4 (usually
0.0.0.0). In this state the bootstrap process hangs as it attempts to
talk and get the mgr state.

A work around is to have ms bind ipv6 = true in a ceph conf
you can then pass to bootstrap which gets pulled in and set in mon's
config store.

This patch sets ms bind ipv6 = true to the global section in the
mon config store when the mon-ip argument is an ipv6 address.

Fixes: https://tracker.ceph.com/issues/45016
Signed-off-by: Matthew Oliver moliver@suse.com

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard backend
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

mgfritch · 2020-06-18T02:26:41Z

src/cephadm/cephadm

+    parser_bootstrap.add_argument(
+        '--ipv6',
+        action='store_true',
+        help='Tell the services to bind to ipv6')


Another approach would be to detect the address family from the
--mon-ip and set appropriately. But that may open up having to dns
resolve hostnames etc.

sadly the ipaddress module doesn't appear to always be present in py2.

but maybe attempt_bind can be modified to detect IPv4/IPv6 ?

>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) >>> s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) >>> s.bind(('::', 1234)) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/socket.py", line 228, in meth return getattr(self._sock,name)(*args) socket.gaierror: [Errno -9] Address family for hostname not supported >>> s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM) >>> s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) >>> s.bind(('0.0.0.0', 1234)) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/socket.py", line 228, in meth return getattr(self._sock,name)(*args) socket.error: [Errno 99] Cannot assign requested address

In either case it might be wise to retain the new --ipv6 argument as a failsafe ... ?

yeah, it would kinda be better if we could have a full dual stack solution too, where services would bind to both ipv4 and 6. But not sure how well that could work or what ramifications that would mean to the messenger code.

I think it might be time for me to fall down that rabbit hole, just out of interest sake. As I think the person bootstrapping should know if they want ipv4 or 6, so --ipv6 is the easy path :)

maybe we can deduce iPv6 by looking at the mon-ip? https://docs.ceph.com/docs/master/cephadm/install/#bootstrap-a-new-cluster

So, either we detect IPv6 here, or we do it in cehp-salt. Having the IPv6 detection here will make things easier for everyone I think.

matthewoliver · 2020-06-19T06:09:45Z

Here is a new version that checks the mon-ip address. It'll failback to not ipv6. It does require the ipaddress module which is builtin from python3.3+. So added a relevent requirements.txt pin and update the tox.ini so the module will be pulled in if < 3.3 (when using tox)

sebastian-philipp · 2020-06-19T07:59:50Z

src/cephadm/cephadm

 import argparse
 import datetime
 import fcntl
+import ipaddress


An idea would be to import this in a try-except block and default to IPv4, if that import fails.

But is it worth it, or can we simply depend on this? I mean, we depend on python >= 3.5 anyway here.

Hm. I think leaving this here is fine.

yeah, I did add ipaddress to the requirements.txt in the cephadm src dir only for py2. If we're only every now >py3.3 then it's a moot point, as is the unicode = str I added to make the code py2 and 3 compliant.

I know ceph is now py3.. so maybe we can clean up the tox and all py2 and py3 checks? But I guess it depends how far back we back port. Octopus is py3 only right, or is that only master (only once we cut the octopus branch)?

sebastian-philipp · 2020-06-19T08:02:27Z

src/cephadm/cephadm

+def is_ipv6(address):
+    # type: (str) -> bool
+    if address.startswith('[') and address.endswith(']'):
+        address = address[1:-1]
+    try:
+        return ipaddress.ip_address(address).version == 6
+    except ValueError:
+        logger.warning("Address: {} isn't a valid IP address".format(address))
+        return False


this function is trivially testable in pytest. Do you want to add something like

def test_is_ipv6(): assert is_ipv6("[::1]") assert not is_ipv6("127.0.0.1")

?

yup, great idea, I'll also add '[]' etc. ta.

I mean non []

If you use cephadm bootstrap with an ipv6 mon ip then currently you'll get into a address family split-brain state, where the mon's messenger connects and binds to ipv6 but the mgr's binds to ipv4 (usually 0.0.0.0). In this state the bootstrap process hangs as it attempts to talk and get the mgr state. A work around is to have `ms bind ipv6 = true` in a ceph conf you can then pass to bootstrap which gets pulled in and set in mon's config store. This patch sets `ms bind ipv6 = true` to the global section in the mon config store when the mon-ip argument is an ipv6 address. Fixes: https://tracker.ceph.com/issues/45016 Signed-off-by: Matthew Oliver <moliver@suse.com>

matthewoliver · 2020-06-19T11:54:10Z

src/cephadm/cephadm

 import argparse
 import datetime
 import fcntl
+import ipaddress


yeah, I did add ipaddress to the requirements.txt in the cephadm src dir only for py2. If we're only every now >py3.3 then it's a moot point, as is the unicode = str I added to make the code py2 and 3 compliant.

I know ceph is now py3.. so maybe we can clean up the tox and all py2 and py3 checks? But I guess it depends how far back we back port. Octopus is py3 only right, or is that only master (only once we cut the octopus branch)?

matthewoliver · 2020-06-19T11:54:59Z

src/cephadm/cephadm


+if sys.version_info > (3, 0):
+    unicode = str
+


this is a horrilbe hack, but allows py2 and 3. And the fact that in py2 the ipaddress needs to be unicode. If this only needs to be py3 only we can loose this.

for now, cephadm is py3 only.

cool, then we should remove py2 from the tox file! It's getting late here and on a Friday, so I'm calling it a night :)

sebastian-philipp · 2020-06-24T12:25:18Z

jenkins test docs

sebastian-philipp · 2020-06-24T13:23:47Z

greenish: http://pulpito.ceph.com/swagner-2020-06-24_11:30:44-rados:cephadm-wip-swagner3-testing-2020-06-24-1025-distro-basic-smithi/

Failues:

https://tracker.ceph.com/issues/46178 (slow ops)
https://tracker.ceph.com/issues/46157 (upgrade)

matthewoliver requested a review from a team as a code owner June 18, 2020 01:57

matthewoliver added the cephadm label Jun 18, 2020

matthewoliver requested review from ricardoasmarques and sebastian-philipp June 18, 2020 01:58

mgfritch reviewed Jun 18, 2020

View reviewed changes

matthewoliver force-pushed the cephadm_bootstap_ipv6 branch from c251aa2 to e26be90 Compare June 19, 2020 06:05

matthewoliver changed the title ~~cephadm: Add ipv6 option to bootstrap~~ cephadm: Set ms bind ipv6 when mon-ip is ipv6 Jun 19, 2020

sebastian-philipp approved these changes Jun 19, 2020

View reviewed changes

matthewoliver force-pushed the cephadm_bootstap_ipv6 branch from e26be90 to 08ba08f Compare June 19, 2020 11:50

matthewoliver commented Jun 19, 2020

View reviewed changes

sebastian-philipp added wip-swagner2-testing wip-swagner3-testing and removed wip-swagner2-testing labels Jun 22, 2020

sebastian-philipp merged commit be09f32 into ceph:master Jun 24, 2020

sebastian-philipp mentioned this pull request Jul 2, 2020

octopus: cephadm batch backport July (1) #35898

Merged

smithfarm mentioned this pull request Jul 23, 2020

Add support for IPv6 SUSE/sesdev#56

Merged

8 tasks

Conversation

matthewoliver commented Jun 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sebastian-philipp Jun 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matthewoliver commented Jun 19, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sebastian-philipp commented Jun 24, 2020

Uh oh!

sebastian-philipp commented Jun 24, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

matthewoliver commented Jun 18, 2020 •

edited

Loading

sebastian-philipp Jun 18, 2020 •

edited

Loading