Elasticsearch version (bin/elasticsearch --version): 6.4.x

Plugins installed: [discovery-file, found-elasticsearch, repository-s3]

JVM version (java -version): 1.8.0_144; 25.144-b01;

OS version (uname -a if on a Unix-like system):Linux 4.4.0-1048-aws #57-Ubuntu SMP

Description of the problem including expected versus actual behavior:
Configuring a s3 repository that uses deprecated credentials specified directly in the repository client settings overwrites the credentials in all repository clients.

Expected behaviour would be that the legacy credentials are only used to populate the settings object that corresponds to the repository/client that actually specified them.

Steps to reproduce:

1. configure a s3 repository (repository 1) that uses secure credentials and a non-default client name
2. verify that this repository works
3. configure a second repository using the default client name and deprecated credentials specified directly in the request creating the repository e.g.

"repository-2": {
    "type": "s3",
    "settings": {
      "bucket": "bucket-name",
      "access_key": "your-access-key",
      "secret_key": "your-secret-key"
    }
  }

4. verify that the repository 2 works
5. find that access to repository 1 is now denied

Additional information:

POST _nodes/reload_secure_settings is a workaround which restores the secure credentials for repository 1 but "breaks" repository 2 in the process.

Screenshot of a heap dump I took from the cluster, see the legacy credentials being propagated into both settings objects

Looking at the s3 repository plugin code the deprecated credentials seem to be propagated into every S3ClientSettings object when the S3 repository object holding the legacy credentials is constructed:

Called here with on settings retrieved from the S3Service object: