Hi @aramprice, I just watched the recording of the foundational infra meeting from April 17th.

To clarify a bit:
Diego is currently using Silk as overlay network that is independent from the IP address(es) of the Diego VM itself.

The idea for IPv6 is to use as much native network routing as possible, i.e. not use an overlay network for IPv6. This means that the IP addresses that would be assigned to the containers will be delegated from an IPv6 prefix that is assigned to the Diego VM.

Your question about whether those IP addresses would move in case of an evacuation: no. These IP addresses are not "sticky" to the app or app instance and would not move.

The goal is to make the networking setup for Diego simpler by using "native" IPv6 addressing. Traffic aimed at a particular container will reach the Diego VM via its CIDR range, and then the Diego VM's kernel can forward the traffic to the container's virtual NIC.

Please also note that the networks are supposed to be dual stack, not pure IPv6. So you would want to assign multiple (at least one IPv4 and one IPv6) networks to the same VM, as @fmoehler mentioned in the call already.

The "prefix" parameter is also the size of the prefix to delegate to each VM from a larger range. Have a look at the discussion while creating the RFC for a more extensive example.

The VM is supposed to self-assign an address to itself. Usually this is the x:y:z::1 address (i.e. the "first" address in the provided CIDR range). Addressing the VM is done via that IP address. The "remaining" addresses are then to do for the VM as it wishes.

Hi @peanball,

Thanks for that additional context - the info and the RFC were helpful.

In thinking about the overall changes to Bosh a few goals or principles (maybe too strong a word) came to mind:

• Make conditional behavior / IP-version specific code and settings as limited as possible
  • I really like that prefix: is used and not ipv6_prefix:
  • Should an IPv4 network allow/require a prefix: (one that selects a single IP) for consistency?
• Reduce the back and forth conversion / formatting where possible (format_ip(), ip.to_i and similar)
  • Ideally a single IpAddrOrCidr (I apologize for the name) or IpAddress representation would be used, and conversions would be handled only where need by the class itself, ex: #to_db_value - for persistence, or #to_s - for logs
• Avoid assumption in the code that bosh will always be dual-stack
  • I realize the system will start out dual-stack but this may not be forever

This isn't to imply disagreement about the current state of things, only to captures my thoughts at the moment.

@aramprice thanks for your feedback!

I just want to elaborate a little on our current idea, but of course this is open for ideas. Regarding your points:

1. We will introduce a "prefix" property in the subnet section of the network (Maybe we can also put it to the network section itself, so that it applies automatically to all subnets?). This property will be considered for ipv4 and ipv6 networks. However if the property is not defined (e.g. for existing networks), the director will consider the "prefix" as /32 (ipv4) or /128 (ipv6) and maintain this accordingly in the database.
2. Yes we will try to get rid of as many conversion as possible. Sometimes it can be handy to have the ips in integer representation, but most of the time we will pass them as IpAddrOrCidr Objects
3. Agree with that. I did not test this (yet), but it might not even be specific for a dual stack setup. Probably it would also need changes in the bosh agent.

Thanks @aramprice, I fully agree with you there. The logical unit of "IP address", with or without netmask (i.e. prefix) should be supported in either scenario and contain the logic of representation.

As @fmoehler mentioned, omitting the prefix will default to a single address (previous behavior).

And there should not be assumptions about dual stack but dual stack should be possible. So far, BOSH supported either v4 or v6, not both at the same time. The mechanism to support more than one network is a classic n+1 problem and can be solved as such where possible.

We just happen to choose n=2 with one v4 and one v6 network. This should be the mindset.

@fmoehler I have a general comment on the term prefix in the config. It's a bit terse and does not convey the meaning very clearly.

In the end this is the prefix (size) delegated to each instance attached to this subnet, right? Can we somehow express this better? My suggestion would be

• delegate_prefix
• delegate_prefix_size
• delegate_cidr_size

alternatively s/delegate/assign/g.

One thing regarding the display of the ip addresses. The bosh-cli is showing the ips as they come from the directors API. As @fmoehler mentioned the prefixes are stored in the db with their prefixes. This also means that the bosh-cli will show the ip addresses with their prefix. Because all the docs are showing examples with ipv4 only this would also mean that everybody has to be prepared that with this change the cli could possibly show the ips (i.e. with command bosh ... vms) in format "ip<v4\v6>/prefix".

One thing regarding the display of the ip addresses. The bosh-cli is showing the ips as they come from the directors API. As @fmoehler mentioned the prefixes are stored in the db with their prefixes. This also means that the bosh-cli will show the ip addresses with their prefix. Because all the docs are showing examples with ipv4 only this would also mean that everybody has to be prepared that with this change the cli could possibly show the ips (i.e. with command bosh ... vms) in format "ip<v4\v6>/prefix".

So far we ensured not to change the API and only return the ip (without the prefix), please refer: https://github.com/cloudfoundry/bosh/pull/2611/files#diff-c9cf4222f4ba33b6b3f35d7f32bfbb2c03a91470e18672bd3dda5468c38a0808R22 so it should not change the way ip addresses are displayed.

Rebased to 282.0.9