firewall status delayed on vcenter #3139
Description
In a vCenter environment, after updating the firewall allowed IP rules on one of the managed hosts the data about the updated firewall rules is delayed in updating for an undetermined amount of time.
The result of this is that vicadmin can show an incorrect firewall status and that VCH create checks will operate based on stale state and incorrectly report success/failure.
This log shows that the VCH deployed at 10.161.23.216 is allowed by the nfsClient firewall rule, even though that specific allowed IP was removed several minutes ago. The firewall is behaving correctly in this case because I can't run/attach a container with the VCH IP not in the allowed IP list.
vicadmin.log
time=2016-11-11T23:10:44.930829084Z level=debug msg=[ END ] [github.com/vmware/vic/lib/install/validate.CreateFromVCHConfig:61] [477.074µs]
time="2016-11-11T23:10:44Z" level=info msg="Using management IP {10.161.23.216 ffffe000} for firewall check"
time="2016-11-11T23:10:44Z" level=info msg="Firewall status: ENABLED on \"/ha-datacenter/host/10.161.22.8/10.161.22.8\""
time="2016-11-11T23:10:44Z" level=debug msg="filtered rules: [{{} nfsClient NFS Client false [{{} 0 65535 outbound dst tcp}] true 0xc4205d1bc0}]"
time="2016-11-11T23:10:44Z" level=debug msg="filtered IPs: [10.162.203.56 10.166.2.116 10.162.202.248 10.162.202.209 10.161.23.216] networks: [] allIP: false rule: nfsClient"
time="2016-11-11T23:10:44Z" level=info msg="Firewall configuration OK on hosts:"
time="2016-11-11T23:10:44Z" level=info msg="\t\"/ha-datacenter/host/10.161.22.8/10.161.22.8\""
This log shows (incorrectly) that host 10.161.23.216 is allowed by firewall rules
vic-machine create
DEBU[2016-11-11T17:25:03-06:00] Network configuration:
DEBU[2016-11-11T17:25:03-06:00] Network: client NetworkEndpoint: &{{ client } true 10.161.23.216/24 {<nil> <nil>} {{ Network:network-18 client } {10.161.23.1 ffffff00} false [] [] []} []}
DEBU[2016-11-11T17:25:03-06:00] Network: management NetworkEndpoint: &{{ } true 10.161.23.216/24 {<nil> <nil>} {{ Network:network-18 management } {10.161.23.1 ffffff00} false [] [] []} []}
DEBU[2016-11-11T17:25:03-06:00] Network: external NetworkEndpoint: &{{ external } true 10.161.23.216/24 {<nil> <nil>} {{ Network:network-18 external } {10.161.23.1 ffffff00} true [] [] []} []}
time=2016-11-11T17:25:03.460107249-06:00 level=debug msg=[BEGIN]
...
[github.com/vmware/vic/lib/install/validate.(*Validator).CheckFirewall:86]
DEBU[2016-11-11T17:25:04-06:00] Checking firewall with management network IP {10.161.23.216 ffffff00}
INFO[2016-11-11T17:25:05-06:00] Firewall status: ENABLED on "/ha-datacenter/host/10.161.22.8/10.161.22.8"
DEBU[2016-11-11T17:25:06-06:00] filtered rules: [{{} nfsClient NFS Client false [{{} 0 65535 outbound dst tcp}] true 0xc420010640}]
DEBU[2016-11-11T17:25:06-06:00] filtered IPs: [10.162.203.56 10.166.2.116 10.162.202.248 10.162.202.209 10.161.23.216] networks: [] allIP: false rule: nfsClient
INFO[2016-11-11T17:25:06-06:00] Firewall configuration OK on hosts:
INFO[2016-11-11T17:25:06-06:00] "/ha-datacenter/host/10.161.22.8/10.161.22.8"