Add a way to configure ClickHouse through YAML #3607
Description
As was discussed on meet-up at 2018.11.15 it would be great to manage configs with not only XML but also with YAML (and json as a proper subset of YAML)
The problem here is that no one of salt, ansible or puppet could serialize data from objects to XML and force to make strange templates like this or (*see internal saltstack jinja2 templates). And with json or YAML, there are out of the box ways to generate configs.
So, basically, for saltstack with pillar structure:
clickhouse:
shards:
cluster_configs:
- cluster: cluster1
shard_user: user1
nodes:
1:
- example01
- example02
2:
- example03
- example04
3:
- example05
- example06
- cluster: cluster2
shard_user: user2
nodes:
1:
- example01
- example02
- example03
2:
- example04
- example05
- example06
I have to create jinja template:
<yandex>
{% if pillar['clickhouse'] is defined -%}
{% if pillar['clickhouse']['shards'] is defined -%}
<remote_servers>
{%- for cluster_config in pillar['clickhouse']['shards']['cluster_configs'] %}
<{{cluster_config['cluster']}}>
{%- for shard in cluster_config['nodes'] %}
<shard>
<internal_replication>true</internal_replication>
{%- for replica in cluster_config['nodes'][shard] %}
<replica>
<host>{{ replica }}</host>
<port>9000</port>
<user>{{cluster_config['shard_user']}}</user>
</replica>
{%- endfor %}
</shard>
{%- endfor %}
</{{cluster_config['cluster']}}>
{%- endfor %}
</remote_servers>
{%- endif %}
{%- endif %}
</yandex>
And instead of it it would be pillar (first iteration):
clickhouse:
clusters:
yandex:
remote_servers:
cluster1:
shard:
- internal_replication: true
replica:
- host: example01
port: 9000
user: user1
- host: example02
port: 9000
user: user1
- internal_replication: true
replica:
- host: example03
port: 9000
user: user1
- host: example04
port: 9000
user: user1
- internal_replication: true
replica:
- host: example05
port: 9000
user: user1
- host: example06
port: 9000
user: user1
cluster2:
shard:
- internal_replication: true
replica:
- host: example01
port: 9000
user: user1
- host: example02
port: 9000
user: user1
- host: example03
port: 9000
user: user1
- internal_replication: true
replica:
- host: example04
port: 9000
user: user1
- host: example05
port: 9000
user: user1
- host: example06
port: 9000
user: user1
and in shards.yaml just:
{% if pillar['clickhouse'] is defined -%}
{% if pillar['clickhouse']['clusters'] is defined -%}
{{ pillar['clikhouse']['clusters']|yaml }}
{%- endif %}
{%- endif %}