What did you do?

I need to remote-write to Amazon Prometheus in an environment where prometheus needs to get SigV$ credentials by making a requests sts-fips.{region}.amazonaws.com

I used to run prometheus v2.55.1

In my kubePrometheusStack helm chart i configured.

prometheusSpec:
  containers:
       - name: prometheus
         env:
           - name: AWS_USE_FIPS_ENDPOINT
             value: "true"

v2.55.1 of prometheus respected this change (Thanks to this PR)

What did you expect to see?

I expected prometheus 3.4.1 to

What did you see instead? Under which circumstances?

After upgrading to prometheus v3.4.1, my prometheus has stopped using sts-fips. Here's the log message

"Failed to apply configuration" err="could not get SigV4 credentials: WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post \"https://sts.us-east-1.amazonaws.com/\"

System information

Linux 5.15.182-123.190.amzn2.x86_64

Prometheus version

version 3.4.1

Prometheus configuration file

remote_write:
- url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/<redacted>/api/v1/remote_write
  sigv4:
    region: us-east-1
    role_arn: <role to assume for remote write>

Alertmanager version

n/a

Alertmanager configuration file

n/a

Logs

"Failed to apply configuration" err="could not get SigV4 credentials: WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post \"https://sts.us-east-1.amazonaws.com/\"