Skip to content

Conversation

@norberttech
Copy link
Member

Change Log

Added

Fixed

  • JsonLoader when writing empty Rows

Changed

  • GoogleSheet rows_in_batch was renamed into rows_per_page

Removed

  • batching logic from extractors

Deprecated

Security


Closes: #719

Description

@github-actions
Copy link
Contributor

github-actions bot commented Nov 4, 2023

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors
+-----------------------+-------------------+------+-----+------------------+-------------------+----------------+
| benchmark             | subject           | revs | its | mem_peak         | mode              | rstdev         |
+-----------------------+-------------------+------+-----+------------------+-------------------+----------------+
| AvroExtractorBench    | bench_extract_10k | 1    | 3   | 44.093mb -20.90% | 436.714ms +22.60% | ±0.47% -35.25% |
| CSVExtractorBench     | bench_extract_10k | 1    | 3   | 13.978mb -65.90% | 344.540ms +29.47% | ±0.26% -86.96% |
| JsonExtractorBench    | bench_extract_10k | 1    | 3   | 18.640mb -73.64% | 694.585ms +23.22% | ±0.19% -29.69% |
| ParquetExtractorBench | bench_extract_10k | 1    | 3   | 242.975mb -3.91% | 1.006s +31.54%    | ±0.08% -91.46% |
| TextExtractorBench    | bench_extract_10k | 1    | 3   | 7.248mb -34.37%  | 21.426ms +49.40%  | ±0.40% -69.29% |
| XmlExtractorBench     | bench_extract_10k | 1    | 3   | 7.595mb -37.31%  | 549.137ms +23.94% | ±0.05% -95.99% |
+-----------------------+-------------------+------+-----+------------------+-------------------+----------------+
Transformers
+-----------------------------+--------------------------+------+-----+-----------------+------------------+----------------+
| benchmark                   | subject                  | revs | its | mem_peak        | mode             | rstdev         |
+-----------------------------+--------------------------+------+-----+-----------------+------------------+----------------+
| RenameEntryTransformerBench | bench_transform_10k_rows | 1    | 3   | 87.033mb +0.00% | 69.143ms +41.87% | ±0.80% +31.15% |
+-----------------------------+--------------------------+------+-----+-----------------+------------------+----------------+
Loaders
+--------------------+----------------+------+-----+------------------+-------------------+----------------+
| benchmark          | subject        | revs | its | mem_peak         | mode              | rstdev         |
+--------------------+----------------+------+-----+------------------+-------------------+----------------+
| AvroLoaderBench    | bench_load_10k | 1    | 3   | 93.197mb -0.00%  | 682.441ms +20.36% | ±2.27% +87.61% |
| CSVLoaderBench     | bench_load_10k | 1    | 3   | 47.111mb -2.41%  | 69.830ms -1.60%   | ±0.53% -82.45% |
| JsonLoaderBench    | bench_load_10k | 1    | 3   | 88.545mb -0.00%  | 78.893ms +38.35%  | ±1.52% -20.04% |
| ParquetLoaderBench | bench_load_10k | 1    | 3   | 286.875mb -1.01% | 1.535s +30.22%    | ±0.54% +94.08% |
| TextLoaderBench    | bench_load_10k | 1    | 3   | 16.532mb -0.00%  | 37.497ms -9.11%   | ±1.25% +83.64% |
+--------------------+----------------+------+-----+------------------+-------------------+----------------+
Building Blocks
+-------------------------+----------------------------+------+-----+-----------------+-------------------+-----------------+
| benchmark               | subject                    | revs | its | mem_peak        | mode              | rstdev          |
+-------------------------+----------------------------+------+-----+-----------------+-------------------+-----------------+
| RowsBench               | bench_chunk_10_on_10k      | 2    | 3   | 60.657mb +0.00% | 4.258ms +92.37%   | ±3.50% +515.30% |
| RowsBench               | bench_diff_left_1k_on_10k  | 2    | 3   | 80.449mb +0.00% | 178.047ms +14.73% | ±0.37% +2.10%   |
| RowsBench               | bench_diff_right_1k_on_10k | 2    | 3   | 58.975mb +0.00% | 18.089ms +18.93%  | ±1.62% +400.98% |
| RowsBench               | bench_drop_1k_on_10k       | 2    | 3   | 59.795mb +0.00% | 2.909ms +59.15%   | ±1.30% -49.97%  |
| RowsBench               | bench_drop_right_1k_on_10k | 2    | 3   | 59.795mb +0.00% | 2.860ms +51.77%   | ±0.84% -62.87%  |
| RowsBench               | bench_entries_on_10k       | 2    | 3   | 59.009mb +0.00% | 4.336ms +69.46%   | ±0.75% -21.34%  |
| RowsBench               | bench_filter_on_10k        | 2    | 3   | 59.538mb +0.00% | 24.980ms +76.61%  | ±1.46% +38.68%  |
| RowsBench               | bench_find_on_10k          | 2    | 3   | 59.537mb +0.00% | 25.054ms +75.01%  | ±1.40% +19.22%  |
| RowsBench               | bench_find_one_on_10k      | 10   | 3   | 57.609mb +0.00% | 2.506μs +39.70%   | ±1.86% -30.26%  |
| RowsBench               | bench_first_on_10k         | 10   | 3   | 57.609mb +0.00% | 0.500μs +66.67%   | ±0.00% 0.00%    |
| RowsBench               | bench_flat_map_on_1k       | 2    | 3   | 65.842mb +0.00% | 14.262ms +34.59%  | ±0.24% -68.96%  |
| RowsBench               | bench_map_on_10k           | 2    | 3   | 91.362mb +0.00% | 64.643ms +31.90%  | ±0.56% -17.10%  |
| RowsBench               | bench_merge_1k_on_10k      | 2    | 3   | 60.058mb +0.00% | 3.345ms +55.04%   | ±1.82% -45.19%  |
| RowsBench               | bench_partition_by_on_10k  | 2    | 3   | 62.328mb +0.00% | 50.488ms +49.19%  | ±1.01% +326.21% |
| RowsBench               | bench_remove_on_10k        | 2    | 3   | 62.159mb +0.00% | 7.941ms +48.92%   | ±1.52% -22.98%  |
| RowsBench               | bench_sort_asc_on_1k       | 2    | 3   | 57.609mb +0.00% | 51.147ms +37.43%  | ±0.33% -70.06%  |
| RowsBench               | bench_sort_by_on_1k        | 2    | 3   | 57.609mb +0.00% | 50.483ms +36.52%  | ±1.56% +398.09% |
| RowsBench               | bench_sort_desc_on_1k      | 2    | 3   | 57.609mb +0.00% | 51.135ms +34.74%  | ±0.93% +22.60%  |
| RowsBench               | bench_sort_entries_on_1k   | 2    | 3   | 59.883mb +0.00% | 9.561ms +31.30%   | ±1.19% +657.60% |
| RowsBench               | bench_sort_on_1k           | 2    | 3   | 57.608mb +0.00% | 37.737ms +31.68%  | ±0.55% -29.54%  |
| RowsBench               | bench_take_1k_on_10k       | 10   | 3   | 57.609mb +0.00% | 21.488μs +61.36%  | ±0.44% -73.12%  |
| RowsBench               | bench_take_right_1k_on_10k | 10   | 3   | 57.609mb +0.00% | 26.601μs +62.80%  | ±1.35% -23.52%  |
| RowsBench               | bench_unique_on_1k         | 2    | 3   | 80.450mb +0.00% | 177.965ms +12.94% | ±0.45% -49.88%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 91.735mb -0.01% | 147.230ms +25.55% | ±0.05% -96.37%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 47.598mb -0.01% | 76.319ms +28.94%  | ±1.56% +82.66%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 12.390mb -0.05% | 17.887ms +24.55%  | ±0.59% -58.40%  |
+-------------------------+----------------------------+------+-----+-----------------+-------------------+-----------------+

@norberttech norberttech merged commit bb8210b into flow-php:1.x Nov 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove batching from Extractors

2 participants