Describe the bug
When creating new namespace observed that ClusterSecretStore is not creating/fetching secret based on refreshInterval (set to 10s) and there is somewhat of delay in creation of secrets (between 4minutes to 7 minutes).
The ExternalSecret manifest is created almost right away but the actual creation of the secret after fetching it from AWS Secret Manager takes some time (please see screenshot) - there is some minutes passing by till the refreshTime is getting populate which indicate the secret was fetch and created locally.

In the ESO log the following log takes the same amount of time till being reported:
{"level":"info","ts":1698923181.4201677,"logger":"controllers.ExternalSecret","msg":"reconciled secret","ExternalSecret":{"name":"artifactory-docker-registry","namespace":"oran-test9"}}

Till then the log is quite stable without any prompt for a change being made:
{"level":"info","ts":1698923177.8425815,"logger":"provider.aws","msg":"using aws session","region":"us-east-1","external id":"","credentials":{}} {"level":"info","ts":1698923177.8426237,"logger":"provider.aws.secretsmanager","msg":"fetching secret map","key":"dev-us-east-1/secret/secret-name1"} {"level":"info","ts":1698923177.8426342,"logger":"provider.aws.secretsmanager","msg":"fetching secret value","key":"dev-us-east-1/secret/secret-name2","version":"AWSCURRENT","value":"SECRET"} {"level":"info","ts":1698923179.8481295,"logger":"provider.aws","msg":"using aws session","region":"us-east-1","external id":"","credentials":{}} {"level":"info","ts":1698923179.8481748,"logger":"provider.aws.secretsmanager","msg":"fetching secret map","key":"dev-us-east-1/secret/secret-name3"} {"level":"info","ts":1698923179.8481805,"logger":"provider.aws.secretsmanager","msg":"fetching secret value","key":"dev-us-east-1/global/playwright-credentials","version":"AWSCURRENT","value":"SECRET"} {"level":"info","ts":1698923180.2435653,"logger":"provider.aws","msg":"using aws session","region":"us-east-1","external id":"","credentials":{}} {"level":"info","ts":1698923180.2436075,"logger":"provider.aws.secretsmanager","msg":"fetching secret map","key":"dev-us-east-1/secret/secret-name4"} {"level":"info","ts":1698923180.2436125,"logger":"provider.aws.secretsmanager","msg":"fetching secret value","key":"dev-us-east-1/mongo/mongo4-user-and-password","version":"AWSCURRENT","value":"SECRET"} {"level":"info","ts":1698923180.4427361,"logger":"provider.aws","msg":"using aws session","region":"us-east-1","external id":"","credentials":{}} {"level":"info","ts":1698923180.44278,"logger":"provider.aws.secretsmanager","msg":"fetching secret map","key":"dev-us-east-1/secret/secret-name5"} {"level":"info","ts":1698923180.4427857,"logger":"provider.aws.secretsmanager","msg":"fetching secret value","key":"dev-us-east-1/global/alertmanager-pagerduty","version":"AWSCURRENT","value":"SECRET"} {"level":"info","ts":1698923180.6423898,"logger":"provider.aws","msg":"using aws session","region":"us-east-1","external id":"","credentials":{}} {"level":"info","ts":1698923180.6424341,"logger":"provider.aws.secretsmanager","msg":"fetching secret map","key":"dev-us-east-1/secret/secret-name6"} {"level":"info","ts":1698923180.6424391,"logger":"provider.aws.secretsmanager","msg":"fetching secret value","key":"dev-us-east-1/secret/secret-name7","version":"AWSCURRENT","value":"SECRET"} {"level":"info","ts":1698923180.844502,"logger":"provider.aws","msg":"using aws session","region":"us-east-1","external id":"","credentials":{}} {"level":"info","ts":1698923180.8445442,"logger":"provider.aws.secretsmanager","msg":"fetching secret map","key":"dev-us-east-1/secret/secret-name8"} {"level":"info","ts":1698923180.844549,"logger":"provider.aws.secretsmanager","msg":"fetching secret value","key":"dev-us-east-1/automation-tests-data-platform","version":"AWSCURRENT","value":"SECRET"}

To Reproduce
Setup ESO v0.9.7 (using helm-chart) as follow:

Create ClusterSecretStore (we are using Terraform for that one) as follow:

We are using ArgoCD and local helm-chart repository to bring all applications under new namespace, which in return create all the manifest - therefore create application which instruct creation of local secrets using ESO (secret provider in our scenario is AWS Secret Manager)

Kubernetes version: EKS 1.26
ESO version: 0.9.7

Expected behavior
Creation of new secrets locally using ESO with fetching from AWS Secret Manager should be based on refreshInterval or any other way as of now it is randomly take place (between 4 to 7 minutes) and create delay in creating new namespace which impact our test automation mechanism.

Screenshots
Attached inline

Additional context
N/A