Skip to content

Commit f6fe0f2

Browse files
al13n321al13n321
committed
Merge remote-tracking branch 'origin/master' into exitstack
2 parents 9812228 + 95a36b9 commit f6fe0f2

230 files changed

Lines changed: 3411 additions & 960 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/en/engines/table-engines/integrations/iceberg.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,14 @@ sidebar_label: Iceberg
66

77
# Iceberg Table Engine
88

9+
:::warning
10+
We recommend using the [Iceberg Table Function](/docs/en/sql-reference/table-functions/iceberg.md) for working with Iceberg data in ClickHouse. The Iceberg Table Function currently provides sufficient functionality, offering a partial read-only interface for Iceberg tables.
11+
12+
The Iceberg Table Engine is available but may have limitations. ClickHouse wasn't originally designed to support tables with externally changing schemas, which can affect the functionality of the Iceberg Table Engine. As a result, some features that work with regular tables may be unavailable or may not function correctly, especially when using the old analyzer.
13+
14+
For optimal compatibility, we suggest using the Iceberg Table Function while we continue to improve support for the Iceberg Table Engine.
15+
:::
16+
917
This engine provides a read-only integration with existing Apache [Iceberg](https://iceberg.apache.org/) tables in Amazon S3, Azure, HDFS and locally stored tables.
1018

1119
## Create Table
@@ -63,6 +71,16 @@ CREATE TABLE iceberg_table ENGINE=IcebergS3(iceberg_conf, filename = 'test_table
6371

6472
Table engine `Iceberg` is an alias to `IcebergS3` now.
6573

74+
**Schema Evolution**
75+
At the moment, with the help of CH, you can read iceberg tables, the schema of which has changed over time. We currently support reading tables where columns have been added and removed, and their order has changed. You can also change a column where a value is required to one where NULL is allowed. Additionally, we support permitted type casting for simple types, namely:  
76+
* int -> long
77+
* float -> double
78+
* decimal(P, S) -> decimal(P', S) where P' > P.
79+
80+
Currently, it is not possible to change nested structures or the types of elements within arrays and maps.
81+
82+
To read a table where the schema has changed after its creation with dynamic schema inference, set allow_dynamic_metadata_for_data_lakes = true when creating the table.
83+
6684
### Data cache {#data-cache}
6785

6886
`Iceberg` table engine and table function support data caching same as `S3`, `AzureBlobStorage`, `HDFS` storages. See [here](../../../engines/table-engines/integrations/s3.md#data-cache).

‎docs/en/engines/table-engines/special/memory.md‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,8 +36,6 @@ Upper and lower bounds can be specified to limit Memory engine table size, effec
3636
- Requires `max_rows_to_keep`
3737
- `max_rows_to_keep` — Maximum rows to keep within memory table where oldest rows are deleted on each insertion (i.e circular buffer). Max rows can exceed the stated limit if the oldest batch of rows to remove falls under the `min_rows_to_keep` limit when adding a large block.
3838
- Default value: `0`
39-
- `compress` - Whether to compress data in memory.
40-
- Default value: `false`
4139

4240
## Usage {#usage}
4341

‎docs/en/sql-reference/functions/date-time-functions.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -195,7 +195,7 @@ makeDateTime64(year, month, day, hour, minute, second[, precision])
195195

196196
**Returned value**
197197

198-
- A date and time created from the supplied arguments. [DateTime64](../../sql-reference/data-types/datetime64.md).
198+
- A date and time created from the supplied arguments. [DateTime64](../../sql-reference/data-types/datetime64.md).
199199

200200
**Example**
201201

‎docs/en/sql-reference/functions/type-conversion-functions.md‎

Lines changed: 51 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -122,7 +122,7 @@ Unsupported arguments:
122122

123123
:::note
124124
If the input value cannot be represented within the bounds of [Int8](../data-types/int-uint.md), overflow or underflow of the result occurs.
125-
This is not considered an error.
125+
This is not considered an error.
126126
For example: `SELECT toInt8(128) == -128;`.
127127
:::
128128

@@ -370,7 +370,7 @@ Unsupported arguments:
370370

371371
:::note
372372
If the input value cannot be represented within the bounds of [Int16](../data-types/int-uint.md), overflow or underflow of the result occurs.
373-
This is not considered an error.
373+
This is not considered an error.
374374
For example: `SELECT toInt16(32768) == -32768;`.
375375
:::
376376

@@ -618,7 +618,7 @@ Unsupported arguments:
618618

619619
:::note
620620
If the input value cannot be represented within the bounds of [Int32](../data-types/int-uint.md), the result over or under flows.
621-
This is not considered an error.
621+
This is not considered an error.
622622
For example: `SELECT toInt32(2147483648) == -2147483648;`
623623
:::
624624

@@ -865,7 +865,7 @@ Unsupported types:
865865

866866
:::note
867867
If the input value cannot be represented within the bounds of [Int64](../data-types/int-uint.md), the result over or under flows.
868-
This is not considered an error.
868+
This is not considered an error.
869869
For example: `SELECT toInt64(9223372036854775808) == -9223372036854775808;`
870870
:::
871871

@@ -1608,7 +1608,7 @@ Unsupported arguments:
16081608

16091609
:::note
16101610
If the input value cannot be represented within the bounds of [UInt8](../data-types/int-uint.md), overflow or underflow of the result occurs.
1611-
This is not considered an error.
1611+
This is not considered an error.
16121612
For example: `SELECT toUInt8(256) == 0;`.
16131613
:::
16141614

@@ -1856,7 +1856,7 @@ Unsupported arguments:
18561856

18571857
:::note
18581858
If the input value cannot be represented within the bounds of [UInt16](../data-types/int-uint.md), overflow or underflow of the result occurs.
1859-
This is not considered an error.
1859+
This is not considered an error.
18601860
For example: `SELECT toUInt16(65536) == 0;`.
18611861
:::
18621862

@@ -2104,7 +2104,7 @@ Unsupported arguments:
21042104

21052105
:::note
21062106
If the input value cannot be represented within the bounds of [UInt32](../data-types/int-uint.md), the result over or under flows.
2107-
This is not considered an error.
2107+
This is not considered an error.
21082108
For example: `SELECT toUInt32(4294967296) == 0;`
21092109
:::
21102110

@@ -2353,7 +2353,7 @@ Unsupported types:
23532353

23542354
:::note
23552355
If the input value cannot be represented within the bounds of [UInt64](../data-types/int-uint.md), the result over or under flows.
2356-
This is not considered an error.
2356+
This is not considered an error.
23572357
For example: `SELECT toUInt64(18446744073709551616) == 0;`
23582358
:::
23592359

@@ -3691,8 +3691,8 @@ toDateTime(expr[, time_zone ])
36913691
- `time_zone` — Time zone. [String](../data-types/string.md).
36923692

36933693
:::note
3694-
If `expr` is a number, it is interpreted as the number of seconds since the beginning of the Unix Epoch (as Unix timestamp).
3695-
If `expr` is a [String](../data-types/string.md), it may be interpreted as a Unix timestamp or as a string representation of date / date with time.
3694+
If `expr` is a number, it is interpreted as the number of seconds since the beginning of the Unix Epoch (as Unix timestamp).
3695+
If `expr` is a [String](../data-types/string.md), it may be interpreted as a Unix timestamp or as a string representation of date / date with time.
36963696
Thus, parsing of short numbers' string representations (up to 4 digits) is explicitly disabled due to ambiguity, e.g. a string `'1999'` may be both a year (an incomplete string representation of Date / DateTime) or a unix timestamp. Longer numeric strings are allowed.
36973697
:::
36983698

@@ -5536,7 +5536,7 @@ Result:
55365536

55375537
## reinterpretAsUInt256
55385538

5539-
Performs byte reinterpretation by treating the input value as a value of type UInt256. Unlike [`CAST`](#cast), the function does not attempt to preserve the original value - if the target type is not able to represent the input type, the output is meaningless.
5539+
Performs byte reinterpretation by treating the input value as a value of type UInt256. Unlike [`CAST`](#cast), the function does not attempt to preserve the original value - if the target type is not able to represent the input type, the output is meaningless.
55405540

55415541
**Syntax**
55425542

@@ -5612,7 +5612,7 @@ Result:
56125612

56135613
## reinterpretAsInt16
56145614

5615-
Performs byte reinterpretation by treating the input value as a value of type Int16. Unlike [`CAST`](#cast), the function does not attempt to preserve the original value - if the target type is not able to represent the input type, the output is meaningless.
5615+
Performs byte reinterpretation by treating the input value as a value of type Int16. Unlike [`CAST`](#cast), the function does not attempt to preserve the original value - if the target type is not able to represent the input type, the output is meaningless.
56165616

56175617
**Syntax**
56185618

@@ -7228,6 +7228,45 @@ Result:
72287228
└──────────────────────────────┘
72297229
```
72307230

7231+
## toUnixTimestamp64Second
7232+
7233+
Converts a `DateTime64` to a `Int64` value with fixed second precision. The input value is scaled up or down appropriately depending on its precision.
7234+
7235+
:::note
7236+
The output value is a timestamp in UTC, not in the timezone of `DateTime64`.
7237+
:::
7238+
7239+
**Syntax**
7240+
7241+
```sql
7242+
toUnixTimestamp64Second(value)
7243+
```
7244+
7245+
**Arguments**
7246+
7247+
- `value` — DateTime64 value with any precision. [DateTime64](../data-types/datetime64.md).
7248+
7249+
**Returned value**
7250+
7251+
- `value` converted to the `Int64` data type. [Int64](../data-types/int-uint.md).
7252+
7253+
**Example**
7254+
7255+
Query:
7256+
7257+
```sql
7258+
WITH toDateTime64('2009-02-13 23:31:31.011', 3, 'UTC') AS dt64
7259+
SELECT toUnixTimestamp64Second(dt64);
7260+
```
7261+
7262+
Result:
7263+
7264+
```response
7265+
┌─toUnixTimestamp64Second(dt64)─┐
7266+
│ 1234567891 │
7267+
└───────────────────────────────┘
7268+
```
7269+
72317270
## toUnixTimestamp64Micro
72327271

72337272
Converts a `DateTime64` to a `Int64` value with fixed microsecond precision. The input value is scaled up or down appropriately depending on its precision.

‎docs/en/sql-reference/statements/create/view.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,7 @@ This feature is deprecated and will be removed in the future.
154154

155155
For your convenience, the old documentation is located [here](https://pastila.nl/?00f32652/fdf07272a7b54bda7e13b919264e449f.md)
156156

157-
## Refreshable Materialized View [Experimental] {#refreshable-materialized-view}
157+
## Refreshable Materialized View {#refreshable-materialized-view}
158158

159159
```sql
160160
CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db.]table_name

‎docs/en/sql-reference/table-functions/iceberg.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,14 @@ SELECT * FROM icebergS3(iceberg_conf, filename = 'test_table')
6565
DESCRIBE icebergS3(iceberg_conf, filename = 'test_table')
6666
```
6767

68+
**Schema Evolution**
69+
At the moment, with the help of CH, you can read iceberg tables, the schema of which has changed over time. We currently support reading tables where columns have been added and removed, and their order has changed. You can also change a column where a value is required to one where NULL is allowed. Additionally, we support permitted type casting for simple types, namely:  
70+
* int -> long
71+
* float -> double
72+
* decimal(P, S) -> decimal(P', S) where P' > P.
73+
74+
Currently, it is not possible to change nested structures or the types of elements within arrays and maps.
75+
6876
**Aliases**
6977

7078
Table function `iceberg` is an alias to `icebergS3` now.

‎programs/disks/CommandRemove.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ class CommandRemove final : public ICommand
3535
throw Exception(ErrorCodes::BAD_ARGUMENTS, "cannot remove '{}': Is a directory", path);
3636
}
3737

38-
disk.getDisk()->removeRecursive(path);
38+
disk.getDisk()->removeRecursiveWithLimit(path);
3939
}
4040
else if (disk.getDisk()->existsFile(path))
4141
{

‎programs/server/Server.cpp‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -296,6 +296,7 @@ namespace CurrentMetrics
296296
extern const Metric MergesMutationsMemoryTracking;
297297
extern const Metric MaxDDLEntryID;
298298
extern const Metric MaxPushedDDLEntryID;
299+
extern const Metric StartupScriptsExecutionState;
299300
}
300301

301302
namespace ProfileEvents
@@ -366,6 +367,14 @@ namespace ErrorCodes
366367
}
367368

368369

370+
enum StartupScriptsExecutionState : CurrentMetrics::Value
371+
{
372+
NotFinished = 0,
373+
Success = 1,
374+
Failure = 2,
375+
};
376+
377+
369378
static std::string getCanonicalPath(std::string && path)
370379
{
371380
Poco::trimInPlace(path);
@@ -782,9 +791,12 @@ void loadStartupScripts(const Poco::Util::AbstractConfiguration & config, Contex
782791
startup_context->makeQueryContext();
783792
executeQuery(read_buffer, write_buffer, true, startup_context, callback, QueryFlags{ .internal = true }, std::nullopt, {});
784793
}
794+
795+
CurrentMetrics::set(CurrentMetrics::StartupScriptsExecutionState, StartupScriptsExecutionState::Success);
785796
}
786797
catch (...)
787798
{
799+
CurrentMetrics::set(CurrentMetrics::StartupScriptsExecutionState, StartupScriptsExecutionState::Failure);
788800
tryLogCurrentException(log, "Failed to parse startup scripts file");
789801
}
790802
}

‎src/AggregateFunctions/Combinators/AggregateFunctionDistinct.h‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,6 @@
88
#include <Interpreters/AggregationCommon.h>
99
#include <Common/HashTable/HashSet.h>
1010
#include <Common/HashTable/HashMap.h>
11-
#include <Common/SipHash.h>
1211
#include <IO/ReadHelpersArena.h>
1312

1413

‎src/Analyzer/ArrayJoinNode.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
#include <Parsers/ASTExpressionList.h>
99
#include <Parsers/ASTTablesInSelectQuery.h>
1010
#include <Common/assert_cast.h>
11+
#include <Common/SipHash.h>
1112

1213
namespace DB
1314
{

0 commit comments

Comments
 (0)