feat: Add Parquet format max_row_group_length option (#19583)

erezrokah · web-flow · commit 4130e1fd449a · 2024-11-08T09:31:40.000Z
#### Summary The support for setting the group length was added in #19578 implicitly via cloudquery/filetypes#589. This PR adds documentation for it.
diff --git a/plugins/destination/azblob/docs/_configuration.md b/plugins/destination/azblob/docs/_configuration.md
@@ -18,6 +18,7 @@ spec:
       # Parquet specific parameters:
       # version: "v2Latest"
       # root_repetition: "repeated"
+      # max_row_group_length: 134217728 # 128 * 1024 * 1024
 
     # Optional parameters
     # compression: "" # options: gzip
diff --git a/plugins/destination/azblob/docs/overview.md b/plugins/destination/azblob/docs/overview.md
@@ -99,3 +99,7 @@ Reserved for future use.
   [Repetition option to use for the root node](https://github.com/apache/arrow/issues/20243). Supported values are `undefined`, `required`, `optional` and `repeated`.
 
   Some Parquet readers require a specific root repetition option to be able to read the file. For example, importing Parquet files into [Snowflake](https://www.snowflake.com/en/) requires the root repetition to be `undefined`.
+
+- `max_row_group_length` (`integer`) (optional) (default: `134217728` (= 128 * 1024 * 1024))
+
+  The maximum number of rows in a single row group. Use a lower number to reduce memory usage when reading the Parquet files, and a higher number to increase the efficiency of reading the Parquet files.
diff --git a/plugins/destination/file/docs/_configuration.md b/plugins/destination/file/docs/_configuration.md
@@ -20,6 +20,7 @@ spec:
       # Parquet specific parameters:
       # version: "v2Latest"
       # root_repetition: "repeated"
+      # max_row_group_length: 134217728 # 128 * 1024 * 1024
     # compression: "" # options: gzip
     # no_rotate: false
     # batch_size: 10000
diff --git a/plugins/destination/file/docs/overview.md b/plugins/destination/file/docs/overview.md
@@ -96,3 +96,7 @@ Reserved for future use.
   [Repetition option to use for the root node](https://github.com/apache/arrow/issues/20243). Supported values are `undefined`, `required`, `optional` and `repeated`.
 
   Some Parquet readers require a specific root repetition option to be able to read the file. For example, importing Parquet files into [Snowflake](https://www.snowflake.com/en/) requires the root repetition to be `undefined`.
+
+- `max_row_group_length` (`integer`) (optional) (default: `134217728` (= 128 * 1024 * 1024))
+
+  The maximum number of rows in a single row group. Use a lower number to reduce memory usage when reading the Parquet files, and a higher number to increase the efficiency of reading the Parquet files.
diff --git a/plugins/destination/gcs/docs/_configuration.md b/plugins/destination/gcs/docs/_configuration.md
@@ -19,6 +19,7 @@ spec:
       # Parquet specific parameters:
       # version: "v2Latest"
       # root_repetition: "repeated"
+      # max_row_group_length: 134217728 # 128 * 1024 * 1024
 
     # Optional parameters
     # compression: "" # options: gzip
diff --git a/plugins/destination/gcs/docs/overview.md b/plugins/destination/gcs/docs/overview.md
@@ -104,6 +104,10 @@ Reserved for future use.
 
   Some Parquet readers require a specific root repetition option to be able to read the file. For example, importing Parquet files into [Snowflake](https://www.snowflake.com/en/) requires the root repetition to be `undefined`.
 
+- `max_row_group_length` (`integer`) (optional) (default: `134217728` (= 128 * 1024 * 1024))
+
+  The maximum number of rows in a single row group. Use a lower number to reduce memory usage when reading the Parquet files, and a higher number to increase the efficiency of reading the Parquet files.
+
 ## Authentication
 
 :authentication
diff --git a/plugins/destination/kafka/docs/_configuration.md b/plugins/destination/kafka/docs/_configuration.md
@@ -24,6 +24,7 @@ spec:
       # Parquet specific parameters:
       # version: "v2Latest"
       # root_repetition: "repeated"
+      # max_row_group_length: 134217728 # 128 * 1024 * 1024
 
     # Optional parameters
     # compression: "" # options: gzip
diff --git a/plugins/destination/kafka/docs/overview.md b/plugins/destination/kafka/docs/overview.md
@@ -89,6 +89,9 @@ Reserved for future use.
 
   Some Parquet readers require a specific root repetition option to be able to read the file. For example, importing Parquet files into [Snowflake](https://www.snowflake.com/en/) requires the root repetition to be `undefined`.
 
+- `max_row_group_length` (`integer`) (optional) (default: `134217728` (= 128 * 1024 * 1024))
+
+  The maximum number of rows in a single row group. Use a lower number to reduce memory usage when reading the Parquet files, and a higher number to increase the efficiency of reading the Parquet files.
 
 ### topic_details
 
diff --git a/plugins/destination/s3/docs/_configuration.md b/plugins/destination/s3/docs/_configuration.md
@@ -23,6 +23,7 @@ spec:
       # Parquet specific parameters:
       # version: "v2Latest"
       # root_repetition: "repeated"
+      # max_row_group_length: 134217728 # 128 * 1024 * 1024
 
     # Optional parameters
     # compression: "" # options: gzip
diff --git a/plugins/destination/s3/docs/overview.md b/plugins/destination/s3/docs/overview.md
@@ -147,6 +147,10 @@ Reserved for future use.
 
   Some Parquet readers require a specific root repetition option to be able to read the file. For example, importing Parquet files into [Snowflake](https://www.snowflake.com/en/) requires the root repetition to be `undefined`.
 
+- `max_row_group_length` (`integer`) (optional) (default: `134217728` (= 128 * 1024 * 1024))
+
+  The maximum number of rows in a single row group. Use a lower number to reduce memory usage when reading the Parquet files, and a higher number to increase the efficiency of reading the Parquet files.
+
 ### server_side_encryption_configuration
 
 - `sse_kms_key_id` (`string`) (required)
