Skip to content

Allow to rename XML nested structure elements on a loader level #1203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
norberttech merged 2 commits into from
Aug 31, 2024
Merged

Allow to rename XML nested structure elements on a loader level #1203

norberttech merged 2 commits into from
Aug 31, 2024

Conversation

@norberttech
Copy link
Member

@norberttech norberttech commented Aug 29, 2024

Change Log

Added

Fixed

Changed

  • Allow to rename XML nested structure elements on a loader level
  • Move optional XMLLoader options to fluent interface

Removed

Deprecated

Security

Description

This change should let us achieve following output:

df()
    ->read(from_array(
        [
            [
                'entry' => [
                    ['g:service' => 'FEDEX', 'g:price' => '12'],
                    ['g:service' => 'UPC', 'g:price' => '10'],
                ]
            ]
        ],
        schema(
            list_schema(
                'entry',
                type_list(
                    struct_type([
                        struct_element('g:service', type_string()),
                        struct_element('g:price', type_string()),
                    ])
                )
            )
        )
    ))
    ->write(to_output(output: Output::schema))
    ->saveMode(overwrite())
    ->write(to_xml(__DIR__ . '/file.xml')->withListElementName('g:shipping'))
    ->run();
<?xml version="1.0" encoding="UTF-8"?>
<rows>
    <row>
        <entry>
            <g:shipping>
                <g:service>FEDEX</g:service>
                <g:price>12</g:price>
            </g:shipping>
            <g:shipping>
                <g:service>UPC</g:service>
                <g:price>10</g:price>
            </g:shipping>
        </entry>
    </row>
</rows>

@github-actions
Copy link
Contributor

github-actions bot commented Aug 29, 2024 •
edited
Loading

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors 
+-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+
| benchmark             | subject           | revs | its | mem_peak        | mode             | rstdev          |
+-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+
| CSVExtractorBench     | bench_extract_10k | 1    | 3   | 4.518mb +0.03%  | 525.763ms +3.16% | ±0.59% +148.80% |
| JsonExtractorBench    | bench_extract_10k | 1    | 3   | 4.585mb +0.03%  | 1.082s -0.68%    | ±0.26% -76.99%  |
| ParquetExtractorBench | bench_extract_10k | 1    | 3   | 29.091mb +0.01% | 433.723ms -0.64% | ±0.33% -61.91%  |
| TextExtractorBench    | bench_extract_10k | 1    | 3   | 4.278mb +0.03%  | 34.387ms +1.17%  | ±0.92% +75.52%  |
| XmlExtractorBench     | bench_extract_10k | 1    | 3   | 4.268mb +0.03%  | 757.076ms +0.31% | ±0.30% -79.42%  |
+-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+
Transformers 
+-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+
| benchmark                   | subject                  | revs | its | mem_peak         | mode            | rstdev         |
+-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+
| RenameEntryTransformerBench | bench_transform_10k_rows | 1    | 3   | 116.554mb +0.00% | 65.166ms +1.92% | ±2.63% +80.03% |
+-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+
Loaders 
+--------------------+----------------+------+-----+------------------+------------------+-----------------+
| benchmark          | subject        | revs | its | mem_peak         | mode             | rstdev          |
+--------------------+----------------+------+-----+------------------+------------------+-----------------+
| CSVLoaderBench     | bench_load_10k | 1    | 3   | 54.714mb +0.00%  | 142.468ms +1.82% | ±1.15% +369.49% |
| JsonLoaderBench    | bench_load_10k | 1    | 3   | 90.327mb +0.00%  | 120.560ms +2.38% | ±0.55% +9.55%   |
| ParquetLoaderBench | bench_load_10k | 1    | 3   | 124.379mb +0.00% | 1.242s +0.48%    | ±0.41% -71.26%  |
| TextLoaderBench    | bench_load_10k | 1    | 3   | 17.469mb +0.01%  | 43.754ms -0.71%  | ±0.49% +55.97%  |
+--------------------+----------------+------+-----+------------------+------------------+-----------------+
Building Blocks 
+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+
| benchmark               | subject                    | revs | its | mem_peak         | mode             | rstdev          |
+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+
| TypeDetectorBench       | bench_type_detector        | 1    | 3   | 53.200mb +0.00%  | 417.061ms +6.83% | ±3.32% +118.52% |
| TypeDetectorBench       | bench_type_detector        | 1    | 3   | 13.466mb +0.01%  | 77.738ms -1.52%  | ±0.72% -68.90%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 107.397mb +0.00% | 476.496ms +0.73% | ±0.84% +15.53%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 55.755mb +0.00%  | 236.746ms -1.53% | ±0.75% -28.35%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 14.593mb +0.01%  | 51.102ms -0.36%  | ±0.16% -94.48%  |
| RowsBench               | bench_chunk_10_on_10k      | 2    | 3   | 87.310mb +0.00%  | 3.436ms -1.56%   | ±1.23% -57.77%  |
| RowsBench               | bench_diff_left_1k_on_10k  | 2    | 3   | 102.914mb +0.00% | 191.106ms -0.89% | ±0.74% -21.35%  |
| RowsBench               | bench_diff_right_1k_on_10k | 2    | 3   | 85.634mb +0.00%  | 18.947ms -1.05%  | ±0.38% -67.80%  |
| RowsBench               | bench_drop_1k_on_10k       | 2    | 3   | 88.550mb +0.00%  | 1.825ms -20.11%  | ±1.05% -40.37%  |
| RowsBench               | bench_drop_right_1k_on_10k | 2    | 3   | 88.550mb +0.00%  | 1.929ms -1.23%   | ±3.74% +46.41%  |
| RowsBench               | bench_entries_on_10k       | 2    | 3   | 85.662mb +0.00%  | 2.924ms -2.04%   | ±1.69% -21.17%  |
| RowsBench               | bench_filter_on_10k        | 2    | 3   | 86.191mb +0.00%  | 16.581ms -10.69% | ±2.18% +59.97%  |
| RowsBench               | bench_find_on_10k          | 2    | 3   | 86.191mb +0.00%  | 17.651ms -6.42%  | ±2.02% +241.31% |
| RowsBench               | bench_find_one_on_10k      | 10   | 3   | 84.095mb +0.00%  | 1.900μs +5.91%   | ±0.00% -100.00% |
| RowsBench               | bench_first_on_10k         | 10   | 3   | 84.095mb +0.00%  | 0.400μs +33.33%  | ±0.00% +0.00%   |
| RowsBench               | bench_flat_map_on_1k       | 2    | 3   | 93.445mb +0.00%  | 12.628ms +0.88%  | ±0.75% -16.21%  |
| RowsBench               | bench_map_on_10k           | 2    | 3   | 122.816mb +0.00% | 65.268ms +4.74%  | ±2.22% +435.38% |
| RowsBench               | bench_merge_1k_on_10k      | 2    | 3   | 86.711mb +0.00%  | 1.558ms +5.66%   | ±1.49% -58.33%  |
| RowsBench               | bench_partition_by_on_10k  | 2    | 3   | 90.067mb +0.00%  | 60.934ms +1.12%  | ±1.50% -10.86%  |
| RowsBench               | bench_remove_on_10k        | 2    | 3   | 88.812mb +0.00%  | 4.374ms +0.31%   | ±1.68% +18.68%  |
| RowsBench               | bench_sort_asc_on_1k       | 2    | 3   | 84.245mb +0.00%  | 39.280ms -0.13%  | ±0.78% -46.90%  |
| RowsBench               | bench_sort_by_on_1k        | 2    | 3   | 84.245mb +0.00%  | 41.588ms +6.38%  | ±1.86% +405.29% |
| RowsBench               | bench_sort_desc_on_1k      | 2    | 3   | 84.245mb +0.00%  | 39.833ms +1.38%  | ±1.47% -7.37%   |
| RowsBench               | bench_sort_entries_on_1k   | 2    | 3   | 86.537mb +0.00%  | 7.373ms -0.12%   | ±0.63% +4.70%   |
| RowsBench               | bench_sort_on_1k           | 2    | 3   | 84.095mb +0.00%  | 29.498ms +3.39%  | ±0.78% -16.87%  |
| RowsBench               | bench_take_1k_on_10k       | 10   | 3   | 84.095mb +0.00%  | 15.138μs +13.46% | ±2.58% +5.97%   |
| RowsBench               | bench_take_right_1k_on_10k | 10   | 3   | 84.095mb +0.00%  | 19.366μs +21.34% | ±1.06% -40.88%  |
| RowsBench               | bench_unique_on_1k         | 2    | 3   | 102.915mb +0.00% | 194.288ms +1.00% | ±0.55% -75.83%  |
+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+

@norberttech
Copy link
Member Author

norberttech commented Aug 30, 2024

@lchrusciel please let me know if that works for your scenario, also if anything is anything is missing don't be shy - I need edge cases 😁

@norberttech
Copy link
Member Author

norberttech commented Aug 31, 2024 •
edited
Loading

It might be beneficial to implement a similar fluent interface on all Loaders/Extractors however it would require a small BC break, please read #1204

@norberttech norberttech merged commit 45c1ad8 into flow-php:1.x Aug 31, 2024
@norberttech norberttech deleted the feature/xml-nested-structure-elements branch December 5, 2024 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

size: S

Projects

Roadmap
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@norberttech