Add EventTransform API types #8447

pierDipi · 2025-02-10T09:16:55Z

Event transformations are essential to connect different systems, with the EventTransform we will support this use cases where a generic and declarative event transformation can be used and we would not require custom code to do that.

The API can be used in different places, between a source/webhook and a broker, between a broker and a sink or even leverage the built-in Trigger transformation feature with response.

source or webhook -> transform -> broker

trigger -> transform -> sink or webhook

broker -> trigger -> (request)  -> transform
broker <- trigger <- (response) <-

Related to:

We would also solve use cases like this https://cloud-native.slack.com/archives/C06AH2C3K8B/p1731786548587349 where a sink is OpenAI compatible LLM endpoint without writing custom code.

First commit is the main commit, rest is codegen and fixes

Proposed Changes

Add EventTransform API
Support for JSON transformations with JSONata expressions: Add Jsonata transformation server knative-extensions/eventing-integrations#62

Pre-review Checklist

At least 80% unit test coverage
E2E tests for any new behavior
Docs PR for any user-facing impact
Spec PR for any new API feature
Conformance test for any change to the spec

Release Note

Add EventTransform API types.

Docs

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

codecov · 2025-02-10T10:34:53Z

Codecov Report

Attention: Patch coverage is 33.33333% with 90 lines in your changes missing coverage. Please review.

Project coverage is 63.92%. Comparing base (4a6e7d2) to head (90e86eb).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...apis/eventing/v1alpha1/eventtransform_lifecycle.go	50.00%	39 Missing and 4 partials ⚠️
...pis/eventing/v1alpha1/eventtransform_validation.go	0.00%	41 Missing ⚠️
pkg/apis/eventing/v1alpha1/eventtransform_types.go	0.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8447      +/-   ##
==========================================
- Coverage   64.10%   63.92%   -0.18%     
==========================================
  Files         388      391       +3     
  Lines       23363    23498     +135     
==========================================
+ Hits        14976    15021      +45     
- Misses       7591     7677      +86     
- Partials      796      800       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

pierDipi · 2025-02-10T11:40:37Z

/test unit-tests

pierDipi · 2025-02-10T12:39:38Z

/test reconciler-tests

pierDipi · 2025-02-10T14:13:19Z

/test reconciler-tests

matzew · 2025-02-10T15:54:20Z

pkg/apis/eventing/v1alpha1/eventtransform_validation.go

+
+func (js *JsonataEventTransformationSpec) Validate(context.Context) *apis.FieldError {
+	// Jsonata parsers for Go are not maintained, therefore, we will not parse the expression here.
+	// The downside is that the errors will only be present in the status of the EventTransform resource.


I think this is not the end of the world, if reflected broken expressions via "runtime errors" propagated to the status of the thing

matzew · 2025-02-10T15:55:19Z

pkg/apis/eventing/v1alpha1/eventtransform_lifecycle_test.go

+        "source": source,
+        "reason": data.reason,
+        "message": data.message,
+        "data": $


has the $ a special meaning?

yes, $ is basically the input / root node

{"hello": "Matthias"} with transformation {"data": $ } results in {"data": {"hello": "Matthias"} }

See for example: https://try.jsonata.org/quwrKXzni

For the same input:

{"data": $.hello, "data2": hello, "full": $ }

returns

{ "data": "Matthias", "data2": "Matthias", "full": { "hello": "Matthias" } }

matzew

/lgtm
/approve

knative-prow · 2025-02-11T08:28:32Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: matzew, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [matzew,pierDipi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* Add EventTransform API Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Codegen Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Rename TransformList -> EventTransformList Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Fix lint errors Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> --------- Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* Add EventTransform API types (knative#8447) * Add EventTransform API Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Codegen Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Rename TransformList -> EventTransformList Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Fix lint errors Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> --------- Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Add EventTransform Jsonata reconciler and CRD (knative#8456) * Add EventTransform Jsonata reconciler Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Add CRD Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Add reconciler to controller binary Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Add permissions Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Add selector to eventing filtered factory Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Add License header Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Add symlink Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> --------- Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * EventTransform: Reconcile address and service (knative#8458) - Set address only once endpoints are available - Add debug logging - Fix semantic comparisons to avoid loops Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * EventTransform: Support transforming response from Sink (knative#8469) This allows: - propagating the response from Sink - discarding the response from Sink - transforming the response from Sink Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * chore: Adding missing sinks and sources to schema file (knative#8460) :lipstick: Adding missing sinks and sources to schema file Signed-off-by: Matthias Wessendorf <mwessend@redhat.com> * Run make generate-release Signed-off-by: Matthias Wessendorf <mwessend@redhat.com> --------- Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> Signed-off-by: Matthias Wessendorf <mwessend@redhat.com> Co-authored-by: Pierangelo Di Pilato <pierdipi@redhat.com>

pierDipi added 2 commits February 10, 2025 10:12

Add EventTransform API

f7c9663

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

Codegen

ab565ce

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

knative-prow bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Feb 10, 2025

knative-prow bot requested review from aslom and creydr February 10, 2025 09:17

knative-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 10, 2025

pierDipi changed the title ~~Add EventTransform API~~ [WIP] Add EventTransform API Feb 10, 2025

knative-prow bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 10, 2025

Rename TransformList -> EventTransformList

56f404a

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

pierDipi force-pushed the eventtransform-api branch from b5c208c to 56f404a Compare February 10, 2025 10:28

Fix lint errors

90e86eb

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

pierDipi changed the title ~~[WIP] Add EventTransform API~~ Add EventTransform API Feb 10, 2025

knative-prow bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 10, 2025

pierDipi changed the title ~~Add EventTransform API~~ Add EventTransform API types Feb 10, 2025

matzew reviewed Feb 10, 2025

View reviewed changes

matzew approved these changes Feb 11, 2025

View reviewed changes

knative-prow bot assigned matzew Feb 11, 2025

knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label Feb 11, 2025

knative-prow bot merged commit 2c1c7b8 into knative:main Feb 11, 2025
35 of 36 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add EventTransform API types #8447

Add EventTransform API types #8447

Uh oh!

pierDipi commented Feb 10, 2025 •

edited

Loading

Uh oh!

codecov bot commented Feb 10, 2025 •

edited

Loading

Uh oh!

pierDipi commented Feb 10, 2025

Uh oh!

pierDipi commented Feb 10, 2025

Uh oh!

pierDipi commented Feb 10, 2025

Uh oh!

matzew Feb 10, 2025

Uh oh!

matzew Feb 10, 2025

Uh oh!

pierDipi Feb 10, 2025 •

edited

Loading

Uh oh!

matzew left a comment

Uh oh!

knative-prow bot commented Feb 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Add EventTransform API types #8447

Add EventTransform API types #8447

Uh oh!

Conversation

pierDipi commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes

Pre-review Checklist

Uh oh!

codecov bot commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pierDipi commented Feb 10, 2025

Uh oh!

pierDipi commented Feb 10, 2025

Uh oh!

pierDipi commented Feb 10, 2025

Uh oh!

matzew Feb 10, 2025

Choose a reason for hiding this comment

Uh oh!

matzew Feb 10, 2025

Choose a reason for hiding this comment

Uh oh!

pierDipi Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matzew left a comment

Choose a reason for hiding this comment

Uh oh!

knative-prow bot commented Feb 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pierDipi commented Feb 10, 2025 •

edited

Loading

codecov bot commented Feb 10, 2025 •

edited

Loading

pierDipi Feb 10, 2025 •

edited

Loading