Skip to content

Conversation

@kacpermuda
Copy link
Contributor

Currently in OpenLineage method we are including all deleted files as datasets which can lead to increasing the size of the event and make matching datasets between jobs harder.

With that change, when using prefixes, we are using them as dataset names and not full file paths. This way, user can easily control the size of the event and also ensure proper matching, when the same two prefixes are passed to different operators. I am also removing the list of files that was saved for the purpose of lineage datasets, introduced in #35838

When reviewing, please take a look at test cases to see how the code will behave now.

Also, I am adjusting prefix typing (hook.list allows list of prefixes) and error raising that was missing in my opinion.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Apr 16, 2024
…in GCSDeleteObjectsOperator

Signed-off-by: Kacper Muda <mudakacper@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants