When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.
Overview of the Issue
When you use the consul connect ca set-config command to update the Vault token the configuration is updated and successfully connects, however when the leadership in the cluster changes or Consul is restarted on a node the vault token in the configuration reverts back to the previous version (as seen by a consul connect ca get-config command.
Reproduction Steps
Steps to reproduce this issue, eg:
- Create a cluster with Vault CA integration
- Revoke the Vault Token and issue a new one
- Update the configuration on Consul with the
consul connect ca set-config command
- Force an election or restart the leader node
It doesn't always happen but I can reproduce this on all of our clusters (DM me if you want remote access to our SBX environment)
Consul info for both Client and Server
Server info
agent:
check_monitors = 0
check_ttls = 0
checks = 3
services = 3
build:
prerelease =
revision = c976ffd2
version = 1.10.3
consul:
acl = enabled
bootstrap = false
known_datacenters = 3
leader = true
leader_addr = 172.16.18.231:8300
server = true
raft:
applied_index = 3459907
commit_index = 3459907
fsm_pending = 0
last_contact = 0
last_log_index = 3459907
last_log_term = 31964
last_snapshot_index = 3459154
last_snapshot_term = 31964
latest_configuration = [{Suffrage:Voter ID:0db759ad-0007-2ea4-4d41-62aff64515d5 Address:172.16.18.231:8300} {Suffrage:Voter ID:4de69dad-f717-cb90-5c23-011fe0301004 Address:172.16.19.63:8300} {Suffrage:Voter ID:b846fa4a-7c4b-f797-33bb-31cd99211e35 Address:172.16.18.7:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 31964
runtime:
arch = amd64
cpu_count = 2
goroutines = 400
max_procs = 2
os = linux
version = go1.16.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 874
failed = 0
health_score = 0
intent_queue = 0
left = 3
member_time = 103391
members = 16
query_queue = 0
query_time = 13
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 6
intent_queue = 0
left = 0
member_time = 15622
members = 9
query_queue = 0
query_time = 1
Operating system and Environment details
Amazon Linux 2 running on EC2 with Consul version 1.10.3 (before upgrade was 1.9.6)
Log Fragments
2021-10-07T00:53:27.206Z [ERROR] agent.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Failed to initialize secondary CA provider: error configuring provider: Error making API request.
URL: GET https://vault.sandbox.homexlabs.com/v1/auth/token/lookup-self
Code: 403. Errors:
* permission denied"
When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.
Overview of the Issue
When you use the
consul connect ca set-configcommand to update the Vault token the configuration is updated and successfully connects, however when the leadership in the cluster changes or Consul is restarted on a node the vault token in the configuration reverts back to the previous version (as seen by aconsul connect ca get-configcommand.Reproduction Steps
Steps to reproduce this issue, eg:
consul connect ca set-configcommandIt doesn't always happen but I can reproduce this on all of our clusters (DM me if you want remote access to our SBX environment)
Consul info for both Client and Server
Server info
Operating system and Environment details
Amazon Linux 2 running on EC2 with Consul version 1.10.3 (before upgrade was 1.9.6)
Log Fragments