Skip to content

AIP-62: add hook lineage collection support for S3Hook #40819

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mobuchowski merged 1 commit into from
Jul 22, 2024
Merged

AIP-62: add hook lineage collection support for S3Hook #40819

mobuchowski merged 1 commit into from
Jul 22, 2024

Conversation

@mobuchowski
Copy link
Contributor

@mobuchowski mobuchowski commented Jul 16, 2024 •
edited
Loading

This PR implements #40819.

This adds support for automatic lineage collection from S3Hook (non-async methods) and adds support for S3 dataset.

@mobuchowski mobuchowski added the AIP-62 Tasks tracking implementation of AIP-62 Getting Lineage from Hook Instrumentation label Jul 16, 2024
@mobuchowski mobuchowski force-pushed the aip-62/s3hook branch 4 times, most recently from 6e2cfb5 to d34b054 Compare July 16, 2024 22:22
@mobuchowski mobuchowski force-pushed the aip-62/s3hook branch 2 times, most recently from 3418daa to b6508e4 Compare July 16, 2024 23:45
This was referenced Jul 16, 2024
Merged
Closed
@mobuchowski mobuchowski force-pushed the aip-62/s3hook branch 2 times, most recently from dbca2ae to c27ffcd Compare July 17, 2024 09:43
potiuk
potiuk reviewed Jul 18, 2024
View reviewed changes
potiuk
potiuk reviewed Jul 18, 2024
View reviewed changes
potiuk
potiuk approved these changes Jul 18, 2024
View reviewed changes
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with one nit about compat dependency.

@kacpermuda kacpermuda mentioned this pull request Jul 19, 2024
Merged
@mobuchowski mobuchowski force-pushed the aip-62/s3hook branch 3 times, most recently from 6dcf0a4 to af69342 Compare July 21, 2024 16:22
@mobuchowski
openlineage: add support for hook lineage for S3Hook
829bdd1 
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
@mobuchowski mobuchowski merged commit 0212f67 into main Jul 22, 2024
@ephraimbuddy ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Jul 22, 2024
@ephraimbuddy ephraimbuddy added this to the Airflow 2.10.0 milestone Jul 23, 2024
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
@mobuchowski @romsharon98
openlineage: add support for hook lineage for S3Hook (apache#40819)
51d3b7f 
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
@eladkal eladkal mentioned this pull request Jul 28, 2024
Closed
31 tasks
@mobuchowski mobuchowski deleted the aip-62/s3hook branch August 2, 2024 13:40
@eladkal eladkal mentioned this pull request Aug 3, 2024
Closed
@kacpermuda kacpermuda mentioned this pull request Sep 26, 2024
Merged
@eladkal eladkal mentioned this pull request Oct 27, 2024
Closed
83 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@potiuk potiuk potiuk approved these changes

@vincbeck vincbeck vincbeck approved these changes

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

@eladkal eladkal Awaiting requested review from eladkal

@o-nikolas o-nikolas Awaiting requested review from o-nikolas

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin

Assignees

No one assigned

Labels

AIP-62 Tasks tracking implementation of AIP-62 Getting Lineage from Hook Instrumentation area:dev-tools area:providers changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) provider:amazon AWS/Amazon - related issues

Projects

None yet

Milestone

Airflow 2.10.0

Development

Successfully merging this pull request may close these issues.

4 participants

@mobuchowski @potiuk @vincbeck @ephraimbuddy