Skip to content

Delayed IAM Role Policy Updates When Using aws_role_arn with DataLakeCatalog #97858

@alsugiliazova

Description

@alsugiliazova

@antonio2368 I am validating a PR related to DataLakeCatalog with aws_role_arn and would like confirmation that I understand the intended behavior correctly.

Setup

  • ClickHouse database created with ENGINE = DataLakeCatalog, catalog_type = 'glue'
  • Access via IAM user that can only sts:AssumeRole
  • Actual permissions come from the assumed role

Expected behavior

When aws_role_arn is specified, all Glue and S3 access should be evaluated strictly using the policies attached to that role, and permission changes on the role should be reflected promptly.

Observed behavior

  • When Glue and S3 permissions are removed from the role, ClickHouse continues to successfully execute SHOW TABLES and queries for several minutes.
  • Permission changes on the role (add/remove Glue or S3 policies) take 3–5 minutes to take effect unless the ClickHouse server is restarted.
  • Restarting the server applies role permission changes immediately.
  • Dropping/recreating the DataLake database does not force permission refresh.
  • In contrast, modifying permissions directly on an IAM user is reflected by ClickHouse in under ~10 seconds.

Questions

  1. Is this delay expected when using aws_role_arn (STS assumed roles)?
  2. Does ClickHouse cache STS credentials or Glue metadata in a way that delays IAM role policy updates?
  3. Is there a recommended way to force permission refresh without restarting the server?

Any guidance on expected behavior or configuration options would be appreciated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions