chore(router): events enhancement for kafka by su-shivanshmathur · Pull Request #8780 · juspay/hyperswitch

su-shivanshmathur · 2025-07-29T06:12:18Z

Type of Change

Description

This feature allows based on deployment config , it is able to query Request Body and add as a top level fields in the Kafka events

Additional Changes

This PR modifies the API contract
This PR modifies the database schema
This PR modifies application configuration/environment variables

Motivation and Context

We wanted a feature where we could add fields from Request Body to top level fields in kafka events irrespective of the flow

How did you test it?

Test 1

In development.toml

[enhancement]
"merchant_reference_id" = "txn_uuid"

{
	"tenant_id": "public",
	"merchant_id": "cloth_seller_kGLq5ZsmKL6hISw8HI9A",
	"api_flow": "PaymentsCreateIntent",
	"created_at_timestamp": 1753767564159,
	"request_id": "019854b1-52c2-7080-ab7f-114e169fc92b",
	"latency": 170,
	"status_code": 200,
	"api_auth_type": "api_key",
	"authentication_data": {
		"merchant_id": "cloth_seller_kGLq5ZsmKL6hISw8HI9A",
		"key_id": "dev_ekHYWGqV5It7fLj62AOa"
	},
	"request": "{\"amount_details\":{\"order_amount\":{\"Value\":100},\"currency\":\"USD\",\"shipping_cost\":null,\"order_tax_amount\":null,\"skip_external_tax_calculation\":\"skip\",\"skip_surcharge_calculation\":\"skip\",\"surcharge_amount\":null,\"tax_on_surcharge\":null},\"merchant_reference_id\":\"NOPA\",\"routing_algorithm_id\":null,\"capture_method\":\"manual\",\"authentication_type\":\"no_three_ds\",\"billing\":{\"address\":{\"city\":\"test\",\"country\":\"NL\",\"line1\":\"*** alloc::string::String ***\",\"line2\":\"*** alloc::string::String ***\",\"line3\":\"*** alloc::string::String ***\",\"zip\":\"*** alloc::string::String ***\",\"state\":\"*** alloc::string::String ***\",\"first_name\":\"*** alloc::string::String ***\",\"last_name\":\"*** alloc::string::String ***\"},\"phone\":{\"number\":\"*** alloc::string::String ***\",\"country_code\":\"+1\"},\"email\":\"*****@example.com\"},\"shipping\":{\"address\":{\"city\":\"Karwar\",\"country\":\"NL\",\"line1\":null,\"line2\":null,\"line3\":null,\"zip\":\"*** alloc::string::String ***\",\"state\":\"*** alloc::string::String ***\",\"first_name\":\"*** alloc::string::String ***\",\"last_name\":\"*** alloc::string::String ***\"},\"phone\":null,\"email\":\"*******@example.com\"},\"customer_id\":\"12345_cus_019841789b727750b33d014ce2fabf10\",\"customer_present\":null,\"description\":null,\"return_url\":null,\"setup_future_usage\":null,\"apply_mit_exemption\":null,\"statement_descriptor\":null,\"order_details\":null,\"allowed_payment_method_types\":null,\"metadata\":null,\"connector_metadata\":null,\"feature_metadata\":null,\"payment_link_enabled\":null,\"payment_link_config\":null,\"request_incremental_authorization\":null,\"session_expiry\":null,\"frm_metadata\":null,\"request_external_three_ds_authentication\":null,\"force_3ds_challenge\":null,\"merchant_connector_details\":null}",
	"user_agent": "PostmanRuntime/7.44.1",
	"ip_addr": "::1",
	"url_path": "/v2/payments/create-intent",
	"response": "{\"id\":\"12345_pay_019854b152cf7bc2b538db792c40bcc6\",\"status\":\"requires_payment_method\",\"amount_details\":{\"order_amount\":100,\"currency\":\"USD\",\"shipping_cost\":null,\"order_tax_amount\":null,\"external_tax_calculation\":\"skip\",\"surcharge_calculation\":\"skip\",\"surcharge_amount\":null,\"tax_on_surcharge\":null},\"client_secret\":\"*** alloc::string::String ***\",\"profile_id\":\"pro_uJBgizwyNrvQmKZVxuUo\",\"merchant_reference_id\":\"NOPA\",\"routing_algorithm_id\":null,\"capture_method\":\"manual\",\"authentication_type\":\"no_three_ds\",\"billing\":{\"address\":{\"city\":\"test\",\"country\":\"NL\",\"line1\":\"*** alloc::string::String ***\",\"line2\":\"*** alloc::string::String ***\",\"line3\":\"*** alloc::string::String ***\",\"zip\":\"*** alloc::string::String ***\",\"state\":\"*** alloc::string::String ***\",\"first_name\":\"*** alloc::string::String ***\",\"last_name\":\"*** alloc::string::String ***\"},\"phone\":{\"number\":\"*** alloc::string::String ***\",\"country_code\":\"+1\"},\"email\":\"*****@example.com\"},\"shipping\":{\"address\":{\"city\":\"Karwar\",\"country\":\"NL\",\"line1\":null,\"line2\":null,\"line3\":null,\"zip\":\"*** alloc::string::String ***\",\"state\":\"*** alloc::string::String ***\",\"first_name\":\"*** alloc::string::String ***\",\"last_name\":\"*** alloc::string::String ***\"},\"phone\":null,\"email\":\"*******@example.com\"},\"customer_id\":\"12345_cus_019841789b727750b33d014ce2fabf10\",\"customer_present\":\"present\",\"description\":null,\"return_url\":null,\"setup_future_usage\":\"on_session\",\"apply_mit_exemption\":\"Skip\",\"statement_descriptor\":null,\"order_details\":null,\"allowed_payment_method_types\":null,\"metadata\":null,\"connector_metadata\":null,\"feature_metadata\":null,\"payment_link_enabled\":\"Skip\",\"payment_link_config\":null,\"request_incremental_authorization\":\"default\",\"expires_on\":\"2025-07-29T05:54:24.111Z\",\"frm_metadata\":null,\"request_external_three_ds_authentication\":\"Skip\"}",
	"error": null,
	"flow_type": "payment",
	"payment_id": "12345_pay_019854b152cf7bc2b538db792c40bcc6",
	"hs_latency": null,
	"http_method": "POST",
	"cluster": "my_own_cluster",
	"txn_uuid": "NOPA"
}

Test 2
In development.toml

[enhancement]
"merchant_reference_id" = "udf_txn_uuid"
"shipping.address.country" = "country"

{
	"tenant_id": "public",
	"merchant_id": "cloth_seller_PdFuIUDbxVwvu7kZJ3DC",
	"api_flow": "PaymentsCreateIntent",
	"created_at_timestamp": 1753795822163,
	"request_id": "01985660-8135-7422-afa3-9c4aa8c4ff23",
	"latency": 264,
	"status_code": 200,
	"api_auth_type": "api_key",
	"authentication_data": {
		"merchant_id": "cloth_seller_PdFuIUDbxVwvu7kZJ3DC",
		"key_id": "dev_SImQYDU2oDTEQ9RqnY4R"
	},
	"request": "{\"amount_details\":{\"order_amount\":{\"Value\":100},\"currency\":\"USD\",\"shipping_cost\":null,\"order_tax_amount\":null,\"skip_external_tax_calculation\":\"skip\",\"skip_surcharge_calculation\":\"skip\",\"surcharge_amount\":null,\"tax_on_surcharge\":null},\"merchant_reference_id\":\"NOPA\",\"routing_algorithm_id\":null,\"capture_method\":\"manual\",\"authentication_type\":\"no_three_ds\",\"billing\":{\"address\":{\"city\":\"test\",\"country\":\"NL\",\"line1\":\"*** alloc::string::String ***\",\"line2\":\"*** alloc::string::String ***\",\"line3\":\"*** alloc::string::String ***\",\"zip\":\"*** alloc::string::String ***\",\"state\":\"*** alloc::string::String ***\",\"first_name\":\"*** alloc::string::String ***\",\"last_name\":\"*** alloc::string::String ***\"},\"phone\":{\"number\":\"*** alloc::string::String ***\",\"country_code\":\"+1\"},\"email\":\"*****@example.com\"},\"shipping\":{\"address\":{\"city\":\"Karwar\",\"country\":\"NL\",\"line1\":null,\"line2\":null,\"line3\":null,\"zip\":\"*** alloc::string::String ***\",\"state\":\"*** alloc::string::String ***\",\"first_name\":\"*** alloc::string::String ***\",\"last_name\":\"*** alloc::string::String ***\"},\"phone\":null,\"email\":\"*******@example.com\"},\"customer_id\":\"12345_cus_019854ddfc4179a1b56936f3ef50cf48\",\"customer_present\":null,\"description\":null,\"return_url\":null,\"setup_future_usage\":null,\"apply_mit_exemption\":null,\"statement_descriptor\":null,\"order_details\":null,\"allowed_payment_method_types\":null,\"metadata\":\"*** serde_json::value::Value ***\",\"connector_metadata\":null,\"feature_metadata\":null,\"payment_link_enabled\":null,\"payment_link_config\":null,\"request_incremental_authorization\":null,\"session_expiry\":null,\"frm_metadata\":null,\"request_external_three_ds_authentication\":null,\"force_3ds_challenge\":null,\"merchant_connector_details\":null}",
	"user_agent": "PostmanRuntime/7.44.1",
	"ip_addr": "::1",
	"url_path": "/v2/payments/create-intent",
	"response": "{\"id\":\"12345_pay_0198566081457b12a4925e86e34cbe96\",\"status\":\"requires_payment_method\",\"amount_details\":{\"order_amount\":100,\"currency\":\"USD\",\"shipping_cost\":null,\"order_tax_amount\":null,\"external_tax_calculation\":\"skip\",\"surcharge_calculation\":\"skip\",\"surcharge_amount\":null,\"tax_on_surcharge\":null},\"client_secret\":\"*** alloc::string::String ***\",\"profile_id\":\"pro_LWkFikerdXPyjg0lghL3\",\"merchant_reference_id\":\"NOPA\",\"routing_algorithm_id\":null,\"capture_method\":\"manual\",\"authentication_type\":\"no_three_ds\",\"billing\":{\"address\":{\"city\":\"test\",\"country\":\"NL\",\"line1\":\"*** alloc::string::String ***\",\"line2\":\"*** alloc::string::String ***\",\"line3\":\"*** alloc::string::String ***\",\"zip\":\"*** alloc::string::String ***\",\"state\":\"*** alloc::string::String ***\",\"first_name\":\"*** alloc::string::String ***\",\"last_name\":\"*** alloc::string::String ***\"},\"phone\":{\"number\":\"*** alloc::string::String ***\",\"country_code\":\"+1\"},\"email\":\"*****@example.com\"},\"shipping\":{\"address\":{\"city\":\"Karwar\",\"country\":\"NL\",\"line1\":null,\"line2\":null,\"line3\":null,\"zip\":\"*** alloc::string::String ***\",\"state\":\"*** alloc::string::String ***\",\"first_name\":\"*** alloc::string::String ***\",\"last_name\":\"*** alloc::string::String ***\"},\"phone\":null,\"email\":\"*******@example.com\"},\"customer_id\":\"12345_cus_019854ddfc4179a1b56936f3ef50cf48\",\"customer_present\":\"present\",\"description\":null,\"return_url\":null,\"setup_future_usage\":\"on_session\",\"apply_mit_exemption\":\"Skip\",\"statement_descriptor\":null,\"order_details\":null,\"allowed_payment_method_types\":null,\"metadata\":\"*** serde_json::value::Value ***\",\"connector_metadata\":null,\"feature_metadata\":null,\"payment_link_enabled\":\"Skip\",\"payment_link_config\":null,\"request_incremental_authorization\":\"default\",\"expires_on\":\"2025-07-29T13:45:22.074Z\",\"frm_metadata\":null,\"request_external_three_ds_authentication\":\"Skip\"}",
	"error": null,
	"flow_type": "payment",
	"payment_id": "12345_pay_0198566081457b12a4925e86e34cbe96",
	"hs_latency": null,
	"http_method": "POST",
	"country": "NL",
	"udf_txn_uuid": "NOPA"
}

Checklist

I formatted the code cargo +nightly fmt --all
I addressed lints thrown by cargo clippy
I reviewed the submitted code
I added unit tests for my changes where possible

semanticdiff-com · 2025-07-29T06:12:21Z

Review changes with

Changed Files

File	Status
crates/router/src/services/api.rs	6% smaller
crates/router/src/configs/secrets_transformers.rs	0% smaller
crates/router/src/configs/settings.rs	0% smaller
crates/router/src/routes/app.rs	0% smaller

…o events_enrichment

hrithikesh026 · 2025-07-29T10:17:58Z

Can you add a test case for retrieving from nested value and list.

…dd-xof-cybersource * 'main' of github.com:juspay/hyperswitch: feat(authentication): added authentication sync api (#8596) feat(connector): [facilitapay] fix refunds, add webhook and void support (#8778) feat(connector): [payload] add recurring payments (#8597) chore(version): 2025.07.31.0 feat(connector): [Flexiti]Add support for flexiti connector (#8743) chore(router): events enhancement for kafka (#8780) ci(cypress): Making a mandate payment with large customer user agents (#8790)

…rver * 'main' of github.com:juspay/hyperswitch: (25 commits) chore: `xof` currency to cybersource cards (#8799) chore(version): 2025.08.01.0 feat(core): Implement UCS based upi for paytm and phonepe (#8732) feat(connector): [katapult]add template code for katapult (#8783) feat(router): introduce `feature` and `feature_data` to gsm (#7771) feat(connector): [cybersource] add changes for field CybersourceConsumerAuthInformation (#8768) feat(authentication): added authentication sync api (#8596) feat(connector): [facilitapay] fix refunds, add webhook and void support (#8778) feat(connector): [payload] add recurring payments (#8597) chore(version): 2025.07.31.0 feat(connector): [Flexiti]Add support for flexiti connector (#8743) chore(router): events enhancement for kafka (#8780) ci(cypress): Making a mandate payment with large customer user agents (#8790) fix(openapi): update create_platform endpoint in api-reference docs (#8782) chore(version): 2025.07.30.0 fix(connector): [GLOBALPAY] Added Tokenization Flow for CITs (#8568) feat(routing): Add api-refs for new decision engine endpoints (#8709) fix: replace xtrim with xdel to support older redis version (#8515) fix(connector): [Worldpay] handle multiple ddc submission for CompleteAuthorize (#8741) feat(connector): [Adyen] receive incoming webhooks for pix expiry (#8720) ...

API events enhancement

1f2e45e

su-shivanshmathur requested review from hrithikesh026 and jarnura July 29, 2025 06:12

su-shivanshmathur requested a review from a team as a code owner July 29, 2025 06:12

hyperswitch-bot bot and others added 3 commits July 29, 2025 06:15

chore: run formatter

c5b41d6

clippy changes

b6696f6

Merge branch 'events_enrichment' of github.com:juspay/hyperswitch int…

040afde

…o events_enrichment

hrithikesh026 approved these changes Jul 29, 2025

View reviewed changes

jarnura approved these changes Jul 29, 2025

View reviewed changes

su-shivanshmathur linked an issue Jul 30, 2025 that may be closed by this pull request

[FEATURE] Request/Response fields to be populated on top level #8804

Closed

2 tasks

Gnanasundari24 added this pull request to the merge queue Jul 30, 2025

Merged via the queue into main with commit e48e55a Jul 30, 2025
21 of 27 checks passed

Gnanasundari24 deleted the events_enrichment branch July 30, 2025 12:01

su-shivanshmathur mentioned this pull request Oct 27, 2025

feat(events): taking request and response both for detail extraction for kafka events #9993

Merged

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(router): events enhancement for kafka#8780

chore(router): events enhancement for kafka#8780
Gnanasundari24 merged 4 commits intomainfrom
events_enrichment

su-shivanshmathur commented Jul 29, 2025 •

edited

Loading

Uh oh!

semanticdiff-com bot commented Jul 29, 2025 •

edited

Loading

Uh oh!

hrithikesh026 commented Jul 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

su-shivanshmathur commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of Change

Description

Additional Changes

Motivation and Context

How did you test it?

Checklist

Uh oh!

semanticdiff-com bot commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hrithikesh026 commented Jul 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

su-shivanshmathur commented Jul 29, 2025 •

edited

Loading

semanticdiff-com bot commented Jul 29, 2025 •

edited

Loading