Introduce support for event-triggered workflows by savingoyal · Pull Request #1271 · Netflix/metaflow

savingoyal · 2023-02-13T07:38:33Z

Items remaining

draft docs - https://github.com/Netflix/metaflow-docs/pull/86/files

romain-intel · 2023-02-13T07:57:17Z

Let me know when this is ready for review. Happy to look and provide feedback whenever you want (I am out on the week of the 19th so responses that week may be delayed).

savingoyal · 2023-02-13T08:44:02Z

Let me know when this is ready for review. Happy to look and provide feedback whenever you want (I am out on the week of the 19th so responses that week may be delayed).

There shouldn't be much to review but I will give you a heads up.

romain-intel · 2023-02-13T08:58:12Z

Sounds good. And yes, I will probably defer to you for any and all argo stuff :). It's just the common decorator(s)/parameter changes that I will review so yes, shouldn't take long. If you get that part done first, I can try to implement internally and see if I hit any snags as well (so we have a bit more confidence that we are all good).

romain-intel

I have not reviewed any of the airflow/argo/step_functions code. I also need to review more the decorators (wrong computer right now to review based on what we currently have). This is therefore an initial high-level review. Happy to talk more.

metaflow/client/core.py

metaflow/events.py

metaflow/client/core.py

metaflow/plugins/__init__.py

metaflow/plugins/argo/argo_events_decorator.py

metaflow/metaflow_config.py

metaflow/plugins/argo/argo_workflows.py

romain-intel

More comments but converging :). I'll try to have the alias thing shortly. Also, please add typing and documentation to all public facing APIs and classes in a manner consistent with the rest of the documentation.

metaflow/events.py

metaflow/client/core.py

metaflow/plugins/argo/argo_events_decorator.py

metaflow/client/core.py

metaflow/events.py

metaflow/plugins/argo/argo_events_decorator.py

metaflow/plugins/argo/argo_workflows_decorator.py

metaflow/plugins/argo/argo_events.py

metaflow/plugins/argo/argo_workflows.py

metaflow/plugins/argo/argo_events_decorator.py

metaflow/plugins/argo/argo_workflows.py

metaflow/plugins/argo/argo_workflows_cli.py

metaflow/plugins/argo/argo_client.py

romain-intel

Not done reviewing yet but initial comments to get them to you as fast as possible.

metaflow/client/core.py

metaflow/events.py

metaflow/plugins/events_decorator.py

metaflow/events.py

metaflow/plugins/argo/argo_client.py

metaflow/plugins/argo/argo_workflows.py

metaflow/plugins/argo/argo_workflows_decorator.py

romain-intel

Ok. I think I am done. I have not reviewed any of the airflow, argo, sfn or kube portions instead focusing on the common changes.

metaflow/events.py

metaflow/plugins/events_decorator.py

metaflow/events.py

* explicitly return None for trigger * address review comments * polishing events docstrings * update event decorator types in docstring

addressed pending comments

savingoyal force-pushed the event-triggering branch from dd318b5 to 69473f8 Compare March 1, 2023 01:18

savingoyal changed the title ~~WIP: Integrate with Argo Events~~ Introduce support for event-triggered workflows Mar 6, 2023

savingoyal mentioned this pull request Mar 6, 2023

User-specified events, event data, and parameter support #1139

Closed

romain-intel previously requested changes Mar 6, 2023

View reviewed changes

savingoyal commented Mar 6, 2023

View reviewed changes

metaflow/metaflow_config.py Outdated Show resolved Hide resolved

jackie-ob reviewed Mar 8, 2023

View reviewed changes

metaflow/plugins/argo/argo_workflows.py Outdated Show resolved Hide resolved

romain-intel reviewed Mar 14, 2023

View reviewed changes

jackie-ob reviewed Mar 15, 2023

View reviewed changes

jackie-ob mentioned this pull request Mar 20, 2023

feature: add delete and terminate for argo workflows #1307

Merged

savingoyal mentioned this pull request Mar 29, 2023

Expand @schedule to trigger based on external events, such as changes to AWS S3 bucket #468

Closed

saikonen reviewed Mar 31, 2023

View reviewed changes

metaflow/plugins/argo/argo_events.py Show resolved Hide resolved

romain-intel added a commit that referenced this pull request Apr 3, 2023

Namespace check skip feature from #1271

415f054

savingoyal pushed a commit that referenced this pull request Apr 3, 2023

Namespace check skip feature from #1271 (#1341)

b27f64c

saikonen reviewed Apr 5, 2023

View reviewed changes

metaflow/plugins/argo/argo_workflows.py Show resolved Hide resolved

metaflow/plugins/argo/argo_workflows.py Show resolved Hide resolved

saikonen reviewed Apr 5, 2023

View reviewed changes

metaflow/plugins/argo/argo_events_decorator.py Outdated Show resolved Hide resolved

metaflow/plugins/argo/argo_workflows.py Outdated Show resolved Hide resolved

metaflow/plugins/argo/argo_workflows.py Outdated Show resolved Hide resolved

savingoyal force-pushed the event-triggering branch from cd3a2bc to 89b13e8 Compare April 24, 2023 02:20

talebzeghmi reviewed Apr 24, 2023

View reviewed changes

metaflow/plugins/argo/argo_workflows_cli.py Show resolved Hide resolved

talebzeghmi reviewed Apr 24, 2023

View reviewed changes

metaflow/plugins/argo/argo_client.py Show resolved Hide resolved

romain-intel reviewed Apr 24, 2023

View reviewed changes

talebzeghmi reviewed Apr 25, 2023

View reviewed changes

metaflow/plugins/argo/argo_client.py Show resolved Hide resolved

saikonen reviewed Apr 25, 2023

View reviewed changes

metaflow/plugins/argo/argo_workflows.py Show resolved Hide resolved

metaflow/plugins/argo/argo_workflows.py Show resolved Hide resolved

saikonen reviewed Apr 25, 2023

View reviewed changes

metaflow/plugins/argo/argo_workflows_decorator.py Show resolved Hide resolved

savingoyal added 5 commits April 25, 2023 09:03

WIP: Integrate with Argo Events

8902299

events

f1fc273

auto-emit events

a104866

address sensor deregistration

fdff141

stuff more information in auto events

6521cf0

savingoyal added 18 commits April 25, 2023 09:06

remove references to AND

896af5b

address feedback for metaflow trigger

8df4f4b

address client changes

8f8117f

add config vars for webhooks

2ca19c1

updates to event triggering

19689f4

more changes

09e8582

changes

3251a67

address feedback

41c59d3

address changes

92df095

add docs

129509e

clean up trigger

a3ef4c5

introduce window

e20ab00

remove hardcodes

36dd508

address pending comments

a5893fe

add todo comments

37977ee

move back event names

ab76cce

clean up code

d2ecb1d

address feedback

62f7089

savingoyal force-pushed the event-triggering branch from 84faed7 to 62f7089 Compare April 25, 2023 16:06

savingoyal added 2 commits April 25, 2023 09:10

support prod namespace

4dee37a

address comments

489f735

romain-intel reviewed Apr 25, 2023

View reviewed changes

address bugs

797aa6d

romain-intel reviewed Apr 25, 2023

View reviewed changes

metaflow/events.py Outdated Show resolved Hide resolved

savingoyal and others added 2 commits April 25, 2023 10:45

final fix

af232ab

fix: event triggering docs (#1372)

df1378e

* explicitly return None for trigger * address review comments * polishing events docstrings * update event decorator types in docstring

savingoyal merged commit daef4fe into master Apr 25, 2023

savingoyal deleted the event-triggering branch April 25, 2023 22:09

saikonen mentioned this pull request Apr 26, 2023

Add labels and fix argo #1360

Merged

Conversation

savingoyal commented Feb 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

romain-intel commented Feb 13, 2023

Uh oh!

savingoyal commented Feb 13, 2023

Uh oh!

romain-intel commented Feb 13, 2023

Uh oh!

romain-intel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

romain-intel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

romain-intel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

romain-intel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

savingoyal commented Feb 13, 2023 •

edited

Loading