Skip to content

Add getEventType() expression function#5686

Merged
dinujoh merged 5 commits intoopensearch-project:mainfrom
ps48:event-type-routing
May 14, 2025
Merged

Add getEventType() expression function#5686
dinujoh merged 5 commits intoopensearch-project:mainfrom
ps48:event-type-routing

Conversation

@ps48
Copy link
Copy Markdown
Member

@ps48 ps48 commented May 7, 2025 •
edited
Loading

Description

Today DataPrepper have a great event metadata structure that users can use to route between sub-pipelines. We already have functions supported in the expression language to help users route these events:

  1. getMetaData- Function to fetch the value of attribute keys from the event
  2. hasTags - Function to validate existence of tags in an event

This PR adds a third metadata related function to help users route based on event Type. getEventType: New function to fetch the event type.

  • Updated Antlr grammar
  • Updated Expressions documentation
  • Updated GenericExpressionIT tests

Example Usage of the new function, the example will be more helpful in the upcoming OTLP source:

otel-logs-pipelines:
  source:
    otel_logs_source:
      ssl: false
      proto_reflection_service: true
  route:
    - logs: "getEventType() == \"LOG\""
  sink:
    - pipeline:
        name: "logs-pipeline"
        routes:
          - "logs"

logs-pipeline:
  source:
    pipeline:
      name: "otel-logs-pipelines"
  sink:
    - stdout:

Issues Resolved

Related issue: #5596
Dependent PR: #5677

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

ps48 added 3 commits May 6, 2025 15:59
@ps48
add geteventType expresion
fc0b4c2 
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
@ps48
update antlr grammar, parser and add expression tests
26e3d80 
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
@ps48
update expression syntax documentation
bc0b866 
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
@ps48
move expression parse exception check to evaluation exceptions
8e2cb29 
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
dlvenable
dlvenable requested changes May 12, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@dlvenable dlvenable left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this improvement @ps48 !

...sion/src/main/java/org/opensearch/dataprepper/expression/GetEventTypeExpressionFunction.java Outdated
}

@Override
public Object evaluate(final List<Object> args, Event event, Function<Object, Object> convertLiteralType) {
Copy link
Copy Markdown
Member

@dlvenable dlvenable May 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add final modifier to all the parameters.

Copy link
Copy Markdown
Member Author

@ps48 ps48 May 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlvenable Sure updated here: 01c342c

...expression/src/main/java/org/opensearch/dataprepper/expression/ParseTreeCoercionService.java Outdated
public ParseTreeCoercionService(final Map<Class<? extends Serializable>, Function<Object, Object>> literalTypeConversions, ExpressionFunctionProvider expressionFunctionProvider) {
public ParseTreeCoercionService(
final Map<Class<? extends Serializable>, Function<Object, Object>> literalTypeConversions,
ExpressionFunctionProvider expressionFunctionProvider) {
Copy link
Copy Markdown
Member

@dlvenable dlvenable May 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a final modifier here.

Copy link
Copy Markdown
Member Author

@ps48 ps48 May 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated: 01c342c

.../src/test/java/org/opensearch/dataprepper/expression/GetEventTypeExpressionFunctionTest.java Outdated
@Test
void testGetEventTypeReturnsCorrectType() {
GetEventTypeExpressionFunction function = createObjectUnderTest();
Event testEvent = createTestEvent("LOG");
Copy link
Copy Markdown
Member

@dlvenable dlvenable May 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this an @ParameterizedTest with multiple event types. e.g. LOG, METRIC, TRACE.

Copy link
Copy Markdown
Member Author

@ps48 ps48 May 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated: 01c342c

@ps48
resolve comments, update getEventTypet test to become parameterized
01c342c 
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
dinujoh
dinujoh reviewed May 12, 2025
View reviewed changes
.../src/test/java/org/opensearch/dataprepper/expression/GetEventTypeExpressionFunctionTest.java
Comment on lines +53 to +58
void testGetEventTypeFunctionRegistered() {
Event testEvent = createTestEvent("event");
final GenericExpressionEvaluator evaluator = mock(GenericExpressionEvaluator.class);
when(evaluator.evaluateConditional("getEventType() == \"event\"", testEvent)).thenReturn(true);
assertDoesNotThrow(() -> evaluator.evaluateConditional("getEventType() == \"event\"", testEvent));
}
Copy link
Copy Markdown
Member

@dinujoh dinujoh May 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this test relevant here ?

Copy link
Copy Markdown
Member Author

@ps48 ps48 May 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dinujoh This function is to check if the function name registration for getEventType is working as expected within expressions. I have a similar one in generic to check actual execution over mock (as done here):

data-prepper/data-prepper-expression/src/test/java/org/opensearch/dataprepper/expression/GenericExpressionEvaluator_ConditionalIT.java

Lines 252 to 253 in 01c342c

arguments("getEventType() == \"event\"", longEvent, true),
arguments("getEventType() == \"LOG\"", longEvent, false)

I feel it is good to have both, but not necessary. Please let me know if want it removed, can do it right away.

@ps48
Copy link
Copy Markdown
Member Author

ps48 commented May 13, 2025

@dinujoh @dlvenable can you please help merge this PR. I don't have permissions to merge this.

@dinujoh dinujoh merged commit a08cd7b into opensearch-project:main May 14, 2025
45 of 47 checks passed
@dlvenable dlvenable added this to the v2.12 milestone May 14, 2025
alparish pushed a commit to alparish/data-prepper that referenced this pull request May 22, 2025
@ps48
Add getEventType() expression function (opensearch-project#5686)
4f8930a 
* add geteventType expresion

Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>

* update antlr grammar, parser and add expression tests

Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>

* update expression syntax documentation

Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>

* move expression parse exception check to evaluation exceptions

Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>

* resolve comments, update getEventTypet test to become parameterized

Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>

---------

Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
@ps48 ps48 mentioned this pull request Sep 24, 2025
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dlvenable dlvenable dlvenable approved these changes

@dinujoh dinujoh dinujoh approved these changes

@sb2k16 sb2k16 Awaiting requested review from sb2k16 sb2k16 is a code owner

@chenqi0805 chenqi0805 Awaiting requested review from chenqi0805

@engechas engechas Awaiting requested review from engechas engechas is a code owner

@san81 san81 Awaiting requested review from san81 san81 is a code owner

@srikanthjg srikanthjg Awaiting requested review from srikanthjg srikanthjg is a code owner

@graytaylor0 graytaylor0 Awaiting requested review from graytaylor0 graytaylor0 is a code owner

@kkondaka kkondaka Awaiting requested review from kkondaka kkondaka is a code owner

@KarstenSchnitter KarstenSchnitter Awaiting requested review from KarstenSchnitter KarstenSchnitter is a code owner

@oeyh oeyh Awaiting requested review from oeyh oeyh is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

v2.12

Development

Successfully merging this pull request may close these issues.

3 participants

@ps48 @dlvenable @dinujoh