Skip to content

It is impossible to override global setting in special section '<clickhouse> <s3>'.  #50029

@CheSema

Description

@CheSema

It is over complicated for and <storage_configuration> sections.

  1. '' and '<storage_configuration> ' has the same settings with different prefixes.

Here is even a comment:

/// ( "max_single_part_upload_size"). Why this happened is a complete mystery to me.

And may be it is OK to have a prefix, but it still inconsistent, see the example:

<clickhouse>
    <s3>
        <broken_s3>
            <endpoint>http://resolver:8083/root/data/</endpoint>
            <check_objects_after_upload>1</check_objects_after_upload>
           <max_connections>1</max_connections>
        </broken_s3>
    </s3>
</clickhouse>

but it must be s3_check_objects_after_upload here:

<clickhouse>
    <storage_configuration>
        <disks>
            <s3>
                <type>s3</type>
                <endpoint>http://minio1:9001/root/data/</endpoint>
                <s3_check_objects_after_upload>1</s3_check_objects_after_upload>
                <s3_max_connections>1</s3_max_connections>
                <max_connections>1</max_connections>
            </s3>
    </storage_configuration>
</clickhouse>

At the same time s3_max_connections is ignored totally, max_connections has effect.

It is just inconvenient. Not a bug at least.

  1. If global settings is set, it unconditionally override corresponding settings from special settings. This is definitely a bug.

The minimal example:

<clickhouse>
    <profiles>
        <default>
            <s3_check_objects_after_upload>1</s3_check_objects_after_upload>
        </default>
    </profiles>
</clickhouse>
<clickhouse>
    <s3>
        <broken_s3>
            <endpoint>http://resolver:8083/root/data/</endpoint>
            <check_objects_after_upload>0</check_objects_after_upload>
        </broken_s3>
    </s3>
</clickhouse>

That happens because CH do that line

request_settings.updateFromSettings(context->getSettings());

It ends up with s3_check_objects_after_upload = 1.

I'm suppose that the original Idea was:

  • apply settings from settings.h at global scope
  • then override them with the settings from stecial config section ' '
  • if the is come query scope settings, apply them and override previous values

In such way it is logic.
For that CH should have to distinguish what have changed between global scope settings and query scope. It doesn't do it for now. Any setting which has value differ from default is marked as changed. Even if the change comes from global scope. So It is impossible to override global setting in special section ' '.

That behaviour comes from the desire to update settings for the client at every read/write. For example diskSettings.h/cpp does not have such problem. Client started once and for a while with a global scope configs. After that query scope settings are passed to the objects like WriteBuffers and take effect how CH write data to the client. But the client doesn't see this settings, actually the is not so much of them which affect the client it self.
With S3 table function and Engine S3 CH tried to solve a difficult issue, how to apply query scope settings to the client, and actually failed with it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions