I do not like this idea.

This feature is related to the iceberg but it brings complexity to the interface which makes other realization overcomplicated.

Lest make the code in Iceberg resolve external links.

I think it is ok in this case

brings complexity to the interface

1. PathWithMetadata is not part of IObjectStorage interface at all -- it's a standalone helper struct. It can even be moved to a standalone header and impl. This structure is perfect for this logic. Adding an optional field doesn't complicate it.
2. The alternative will be way messier -- juggling <path, storage> tuples everywhere, doing repeated lookups for every file, introducing some cache that will resolve path into object storage and key, or even introducing a struct that will be almost similar and serve for the same purpose. Neither of these will help make it less complicated. Encapsulating it here is clean and natural.
3. The new logic in this struct is completely optional -- in other places (where we don't need it) it can be used old way, existing code is unaffected.

It looks like you know what to do: p.2.

My opinion that 1k rows for the particular feature in the designated place it much better then 1k rows scattered every where which you have to ignore in the most cases.

Of course, but here we do not have that "ideal" case: the intersection of Iceberg and generic object storage logic and other server logic is large, it is not always encapsulated. There are already many places in code where we have a generic workflow but have specific branching for Iceberg, e.g. like this simple one. Iceberg module does not work with object storages and data files at lower level, that logic remains outside Iceberg.

Current code is actually quite isolated (not 100%, but maybe 80%). Most of the changes outside Iceberg territory is renaming RelativePathWithMetadata -> PathWithMetadata (and also corresponding variables), which does not in any way worsen the code itself. There is also passing additional variables to methods. Now they are only used in Iceberg workflow, but if at some point we need similar multi-storage data lake logic somewhere else -- they will be useful (I do not know whether it will or won't happen, but it may). Sometimes these arguments are passed and checked in an if -- which does not affect other code that does not pass those.

tl;dr: Sure, this code can be improved to be isolated better. This would involve a lot of changes to current structure, but I am not sure this is justified. Iceberg module heavily delegates actual data reading and interaction with object storages outside (which is IMO totally right), to our usual reading-from-object-storage classes, iterators and methods.

I'm agree with @CheSema, let's cleanup this

I'm agree with @CheSema, let's cleanup this

Secondary sounds strange in this case. How do you understand that object storage is primary or secondary?

And why don't we need to put "primary" storage to this map with secondary storages?

Because of the logic itself:

We have our "main" object storage, where we read from by default. But it may happen that we stumble upon a file that belongs to another storage -- thus that storage will be socondary.

file:// paths are currently constrained to user_files, but this check can be bypassed through symlinks: ensure_local_path_inside_user_files validates the target with lexically_normal, while the created LocalObjectStorage later reads remote_path directly (createReadBufferFromFileBase) without resolving symlinks.

A concrete trace:

1. user_files = /var/lib/clickhouse/user_files.
2. Metadata path = file:///var/lib/clickhouse/user_files/link/secret.parquet, where link -> /etc.
3. ensure_local_path_inside_user_files sees lexical path /var/lib/clickhouse/user_files/link/secret.parquet and allows it.
4. LocalObjectStorage::readObject opens /var/lib/clickhouse/user_files/link/secret.parquet, which dereferences to /etc/secret.parquet.

So untrusted metadata can still escape user_files via symlink traversal.

Can we harden resolveObjectStorageForPath by validating with canonicalized paths (resolving symlinks), or by rejecting any path component that is a symlink before opening?

this will be secured in #99405. no need to invent bicycles here.