Skip to content

feat(opensearch): add amount and customer_id as filters and handle name for different indexes#7073

Merged
likhinbopanna merged 8 commits intomainfrom
amount-filter-global-search
Feb 7, 2025
Merged

feat(opensearch): add amount and customer_id as filters and handle name for different indexes#7073
likhinbopanna merged 8 commits intomainfrom
amount-filter-global-search

Conversation

@tsdk02
Copy link
Contributor

@tsdk02 tsdk02 commented Jan 20, 2025

Type of Change

  • Bugfix
  • New feature
  • Enhancement
  • Refactoring
  • Dependency updates
  • Documentation
  • CI/CD

Description

amount filter

  • Added amount as a filter / category for global search.
  • Payments / Refunds / Disputes can now be searched in a better manner by applying an amount filter in the following - format:
amount:100
  • Handling of the field name while building the opensearch query is done.
    • PaymentAttempts / SessionizerPaymentAttempts / PaymentIntents / SessionizerPaymentIntents - amount
    • Refunds / SessionizerRefunds - refund_amount
    • Disputes / SessionizerDisputes - dispute_amount

customer_id filter

  • Also added customer_id as a filter.
  • This will enable searching and filtering customers based on the customer_id filter on the Customers page, which uses global search.

Additional Changes

  • This PR modifies the API contract
  • This PR modifies the database schema
  • This PR modifies application configuration/environment variables

Motivation and Context

For better search experience based on filters using global search and also on the Customers page.

How did you test it?

amount filter:

  • Hit the curl:
curl --location 'http://localhost:8080/analytics/v1/search' \
--header 'sec-ch-ua-platform: "macOS"' \
--header 'authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoiNmNkZTA0NzYtMTFlMi00NGE5LTlkMjUtOTA5M2QzNDQwZjhlIiwibWVyY2hhbnRfaWQiOiJtZXJjaGFudF8xNzM1MDQxMjkzIiwicm9sZV9pZCI6Im9yZ19hZG1pbiIsImV4cCI6MTczNzUzMjA3OSwib3JnX2lkIjoib3JnX2pwanI5TkFEWlhqSENNYTU5MmF3IiwicHJvZmlsZV9pZCI6InByb19QRHUydVA3aWNuM2lXY0I3V0VVSSIsInRlbmFudF9pZCI6InB1YmxpYyJ9.1DAbjUFNOCjpZM_K_NlKj6hOvzP28xLvl01XACE4jPg' \
--header 'Referer: http://localhost:9000/' \
--header 'sec-ch-ua: "Google Chrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'api-key: dev_ssyiqBXK7Ou1HUQuH0Z80BsHuV5pojc0uWli4QbNXGM1f4pn7jCIyb9VFiHlyATQ' \
--header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36' \
--header 'Content-Type: application/json' \
--data '{
    "query": "usd",
    "filters": {
        "amount": [
            600
        ]
    }
}'
  • You can see the amount filter getting applied, and should get results after the amount filter is applied.

  • Sample opensearch query structure:

    • Sessionizer Payment Intents Index: (Notice the amount field in the payload)
Index: SessionizerPaymentIntents
Payload: {
  "query": {
    "bool": {
      "filter": [
        {
          "multi_match": {
            "type": "phrase",
            "query": "usd",
            "lenient": true
          }
        },
        {
          "terms": {
            "amount": [
              600
            ]
          }
        }
      ],
      "must": [
        {
          "bool": {
            "must": [
              {
                "bool": {
                  "should": [
                    {
                      "bool": {
                        "must": [
                          {
                            "term": {
                              "organization_id.keyword": {
                                "value": "org_jpjr9NADZXjHCMa592aw"
                              }
                            }
                          }
                        ]
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              }
            ]
          }
        }
      ]
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}
  • Sessionizer Refunds Index: (Notice the refund_amount being used here)
Index: SessionizerRefunds
Payload: {
  "query": {
    "bool": {
      "filter": [
        {
          "multi_match": {
            "type": "phrase",
            "query": "usd",
            "lenient": true
          }
        },
        {
          "terms": {
            "refund_amount": [
              600
            ]
          }
        }
      ],
      "must": [
        {
          "bool": {
            "must": [
              {
                "bool": {
                  "should": [
                    {
                      "bool": {
                        "must": [
                          {
                            "term": {
                              "organization_id.keyword": {
                                "value": "org_jpjr9NADZXjHCMa592aw"
                              }
                            }
                          }
                        ]
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              }
            ]
          }
        }
      ]
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}
  • Sample response:
[
    {
        "count": 0,
        "index": "payment_attempts",
        "hits": [],
        "status": "Success"
    },
    {
        "count": 0,
        "index": "payment_intents",
        "hits": [],
        "status": "Success"
    },
    {
        "count": 1,
        "index": "refunds",
        "hits": [
            {
                "@timestamp": "2025-01-20T13:02:17Z",
                "attempt_id": "pay_ND5OdiSMSt6lhWovQup1_1",
                "connector": "stripe_test",
                "connector_refund_id": "dummy_ref_isNEfXmc0SW2r0cvaHvc",
                "connector_transaction_id": "pay_dWLQFQ9O5CcNP73haAqF",
                "created_at": 1737378137,
                "currency": "USD",
                "description": "Customer returned product",
                "external_reference_id": "ref_VIVDwLPl0zwwrRSJtnMP",
                "headers": {},
                "internal_reference_id": "refid_EqKX3XCwLUAw9vd8bk9Z",
                "merchant_id": "merchant_1735041293",
                "modified_at": 1737378138,
                "organization_id": "org_jpjr9NADZXjHCMa592aw",
                "payment_id": "pay_ND5OdiSMSt6lhWovQup1",
                "profile_id": "pro_PDu2uP7icn3iWcB7WEUI",
                "refund_amount": 600,
                "refund_arn": "",
                "refund_error_code": null,
                "refund_error_message": null,
                "refund_id": "ref_VIVDwLPl0zwwrRSJtnMP",
                "refund_reason": "Customer returned product",
                "refund_status": "success",
                "refund_type": "instant_refund",
                "sent_to_gateway": true,
                "source_type": "kafka",
                "tenant_id": "public",
                "timestamp": "2025-01-20T13:02:17Z",
                "total_amount": 10000
            }
        ],
        "status": "Success"
    },
    {
        "count": 0,
        "index": "disputes",
        "hits": [],
        "status": "Failure"
    },
    {
        "count": 0,
        "index": "sessionizer_payment_attempts",
        "hits": [],
        "status": "Failure"
    },
    {
        "count": 0,
        "index": "sessionizer_payment_intents",
        "hits": [],
        "status": "Failure"
    },
    {
        "count": 0,
        "index": "sessionizer_refunds",
        "hits": [],
        "status": "Failure"
    },
    {
        "count": 0,
        "index": "sessionizer_disputes",
        "hits": [],
        "status": "Failure"
    }
]

customer_id filter:

  • Hit the curl:
curl --location 'http://localhost:8080/analytics/v1/search' \
--header 'sec-ch-ua-platform: "macOS"' \
--header 'authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoiNmNkZTA0NzYtMTFlMi00NGE5LTlkMjUtOTA5M2QzNDQwZjhlIiwibWVyY2hhbnRfaWQiOiJtZXJjaGFudF8xNzM1MDQxMjkzIiwicm9sZV9pZCI6Im9yZ19hZG1pbiIsImV4cCI6MTczODg0MDA2MCwib3JnX2lkIjoib3JnX2pwanI5TkFEWlhqSENNYTU5MmF3IiwicHJvZmlsZV9pZCI6InByb19QRHUydVA3aWNuM2lXY0I3V0VVSSIsInRlbmFudF9pZCI6InB1YmxpYyJ9.frlVM-TVh88J4dxfJgsMoQv5DdLnD6b-qTTr_DIU90s' \
--header 'Referer: http://localhost:9000/' \
--header 'sec-ch-ua: "Google Chrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'api-key: dev_ssyiqBXK7Ou1HUQuH0Z80BsHuV5pojc0uWli4QbNXGM1f4pn7jCIyb9VFiHlyATQ' \
--header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36' \
--header 'Content-Type: application/json' \
--data '{
    "query": "usd",
    "filters": {
        "customer_id": [
            "StripeCustomer"
        ]
    }
}'
  • Sample Resposne:
[
    {
        "count": 0,
        "index": "payment_attempts",
        "hits": [],
        "status": "Success"
    },
    {
        "count": 6,
        "index": "payment_intents",
        "hits": [
            {
                "@timestamp": "2025-01-20T13:01:37Z",
                "active_attempt_id": "pay_ND5OdiSMSt6lhWovQup1_1",
                "amount": 10000,
                "amount_captured": 10000,
                "attempt_count": 1,
                "billing_details": null,
                "business_country": null,
                "business_label": "default",
                "client_secret": "pay_ND5OdiSMSt6lhWovQup1_secret_TWKqD8yqhgbyEXowOSe3",
                "connector_id": null,
                "created_at": 1737378097,
                "currency": "USD",
                "customer_email": "f6cd0753a831880835a0e0d23136ef66318d8303605bd37fa80b4ecf248bd303",
                "customer_id": "StripeCustomer",
                "description": "Its my first payment request",
                "feature_metadata": {
                    "apple_pay_recurring_details": null,
                    "redirect_response": null,
                    "search_tags": [
                        "1259195bb05bb44ea78ab60b26a54065183f91b3c5b3c2c074cadcf521305a79",
                        "9bff7b14d0a57a00d98f5eb742418a50b3bdfe5993f16b4219bfa31989bdf80f"
                    ]
                },
                "headers": {},
                "merchant_id": "merchant_1735041293",
                "merchant_order_reference_id": null,
                "metadata": "{\"data2\":\"camel\",\"login_date\":\"2019-09-10T10:11:12Z\",\"new_customer\":\"true\"}",
                "modified_at": 1737378099,
                "off_session": null,
                "organization_id": "org_jpjr9NADZXjHCMa592aw",
                "payment_confirm_source": null,
                "payment_id": "pay_ND5OdiSMSt6lhWovQup1",
                "profile_id": "pro_PDu2uP7icn3iWcB7WEUI",
                "return_url": "https://google.com/",
                "setup_future_usage": null,
                "shipping_details": null,
                "source_type": "kafka",
                "statement_descriptor_name": "joseph",
                "statement_descriptor_suffix": "JS",
                "status": "succeeded",
                "tenant_id": "public",
                "timestamp": "2025-01-20T13:01:37Z"
            },
            {
                "@timestamp": "2025-01-20T09:56:07Z",
                "active_attempt_id": "pay_bWGMDT130V4mrDskx9Lt_1",
                "amount": 6540,
                "amount_captured": 6540,
                "attempt_count": 1,
                "billing_details": null,
                "business_country": null,
                "business_label": "default",
                "client_secret": "pay_bWGMDT130V4mrDskx9Lt_secret_VnTfCd82o07H59gQOmdA",
                "connector_id": null,
                "created_at": 1737366967,
                "currency": "USD",
                "customer_email": "6ede5d9641558aef42ea1cff12554f6d6f56081319465af002f0842aec34c763",
                "customer_id": "StripeCustomer",
                "description": "Its my first payment request",
                "feature_metadata": null,
                "headers": {},
                "merchant_id": "merchant_1735041293",
                "merchant_order_reference_id": null,
                "metadata": "{\"udf1\":\"value1\",\"login_date\":\"2019-09-10T10:11:12Z\",\"new_customer\":\"true\"}",
                "modified_at": 1737366971,
                "off_session": null,
                "organization_id": "org_jpjr9NADZXjHCMa592aw",
                "payment_confirm_source": null,
                "payment_id": "pay_bWGMDT130V4mrDskx9Lt",
                "profile_id": "pro_PDu2uP7icn3iWcB7WEUI",
                "return_url": "https://google.com/",
                "setup_future_usage": null,
                "shipping_details": null,
                "source_type": "kafka",
                "statement_descriptor_name": "joseph",
                "statement_descriptor_suffix": "JS",
                "status": "succeeded",
                "tenant_id": "public",
                "timestamp": "2025-01-20T09:56:07Z"
            },
            {
                "@timestamp": "2025-01-20T09:50:20Z",
                "active_attempt_id": "pay_OEYLy5ZKarSVBVOyhsON_1",
                "amount": 6540,
                "amount_captured": 6540,
                "attempt_count": 1,
                "billing_details": null,
                "business_country": null,
                "business_label": "default",
                "client_secret": "pay_OEYLy5ZKarSVBVOyhsON_secret_EmNPjhoI9FK8aVO30Ss4",
                "connector_id": null,
                "created_at": 1737366620,
                "currency": "USD",
                "customer_email": "6ede5d9641558aef42ea1cff12554f6d6f56081319465af002f0842aec34c763",
                "customer_id": "StripeCustomer",
                "description": "Its my first payment request",
                "feature_metadata": null,
                "headers": {},
                "merchant_id": "merchant_1735041293",
                "merchant_order_reference_id": null,
                "metadata": "{\"udf1\":\"value1\",\"login_date\":\"2019-09-10T10:11:12Z\",\"new_customer\":\"true\"}",
                "modified_at": 1737366627,
                "off_session": null,
                "organization_id": "org_jpjr9NADZXjHCMa592aw",
                "payment_confirm_source": null,
                "payment_id": "pay_OEYLy5ZKarSVBVOyhsON",
                "profile_id": "pro_PDu2uP7icn3iWcB7WEUI",
                "return_url": "https://google.com/",
                "setup_future_usage": null,
                "shipping_details": null,
                "source_type": "kafka",
                "statement_descriptor_name": "joseph",
                "statement_descriptor_suffix": "JS",
                "status": "succeeded",
                "tenant_id": "public",
                "timestamp": "2025-01-20T09:50:20Z"
            },
            {
                "@timestamp": "2025-01-20T09:41:41Z",
                "active_attempt_id": "pay_lJJCj6jGDPPOtwb3YucZ_1",
                "amount": 10000,
                "amount_captured": 10000,
                "attempt_count": 1,
                "billing_details": null,
                "business_country": null,
                "business_label": "default",
                "client_secret": "pay_lJJCj6jGDPPOtwb3YucZ_secret_xDqPGN0uREz1mSDLeQv9",
                "connector_id": null,
                "created_at": 1737366101,
                "currency": "USD",
                "customer_email": "f6cd0753a831880835a0e0d23136ef66318d8303605bd37fa80b4ecf248bd303",
                "customer_id": "StripeCustomer",
                "description": "Its my first payment request",
                "feature_metadata": {
                    "apple_pay_recurring_details": null,
                    "redirect_response": null,
                    "search_tags": [
                        "1259195bb05bb44ea78ab60b26a54065183f91b3c5b3c2c074cadcf521305a79",
                        "9bff7b14d0a57a00d98f5eb742418a50b3bdfe5993f16b4219bfa31989bdf80f"
                    ]
                },
                "headers": {},
                "merchant_id": "merchant_1735041293",
                "merchant_order_reference_id": null,
                "metadata": "{\"data2\":\"camel\",\"login_date\":\"2019-09-10T10:11:12Z\",\"new_customer\":\"true\"}",
                "modified_at": 1737366102,
                "off_session": null,
                "organization_id": "org_jpjr9NADZXjHCMa592aw",
                "payment_confirm_source": null,
                "payment_id": "pay_lJJCj6jGDPPOtwb3YucZ",
                "profile_id": "pro_PDu2uP7icn3iWcB7WEUI",
                "return_url": "https://google.com/",
                "setup_future_usage": null,
                "shipping_details": null,
                "source_type": "kafka",
                "statement_descriptor_name": "joseph",
                "statement_descriptor_suffix": "JS",
                "status": "succeeded",
                "tenant_id": "public",
                "timestamp": "2025-01-20T09:41:41Z"
            },
            {
                "@timestamp": "2025-01-20T08:13:08Z",
                "active_attempt_id": "pay_A3dPrR94eHFTlvT7CSXT_1",
                "amount": 10000,
                "amount_captured": 10000,
                "attempt_count": 1,
                "billing_details": null,
                "business_country": null,
                "business_label": "default",
                "client_secret": "pay_A3dPrR94eHFTlvT7CSXT_secret_3FUMXfThPXlRp0kZyvPc",
                "connector_id": null,
                "created_at": 1737360788,
                "currency": "USD",
                "customer_email": "f6cd0753a831880835a0e0d23136ef66318d8303605bd37fa80b4ecf248bd303",
                "customer_id": "StripeCustomer",
                "description": "Its my first payment request",
                "feature_metadata": {
                    "apple_pay_recurring_details": null,
                    "redirect_response": null,
                    "search_tags": [
                        "1259195bb05bb44ea78ab60b26a54065183f91b3c5b3c2c074cadcf521305a79",
                        "9bff7b14d0a57a00d98f5eb742418a50b3bdfe5993f16b4219bfa31989bdf80f"
                    ]
                },
                "headers": {},
                "merchant_id": "merchant_1735041293",
                "merchant_order_reference_id": null,
                "metadata": "{\"data2\":\"camel\",\"login_date\":\"2019-09-10T10:11:12Z\",\"new_customer\":\"true\"}",
                "modified_at": 1737360790,
                "off_session": null,
                "organization_id": "org_jpjr9NADZXjHCMa592aw",
                "payment_confirm_source": null,
                "payment_id": "pay_A3dPrR94eHFTlvT7CSXT",
                "profile_id": "pro_PDu2uP7icn3iWcB7WEUI",
                "return_url": "https://google.com/",
                "setup_future_usage": null,
                "shipping_details": null,
                "source_type": "kafka",
                "statement_descriptor_name": "joseph",
                "statement_descriptor_suffix": "JS",
                "status": "succeeded",
                "tenant_id": "public",
                "timestamp": "2025-01-20T08:13:08Z"
            },
            {
                "@timestamp": "2024-12-24T11:56:12Z",
                "active_attempt_id": "pay_5cvsiaRXZRzRg6VNvJ5e_1",
                "amount": 10000,
                "amount_captured": 10000,
                "attempt_count": 1,
                "billing_details": null,
                "business_country": null,
                "business_label": "default",
                "client_secret": "pay_5cvsiaRXZRzRg6VNvJ5e_secret_XRpkHJHWahWqIaYKiovr",
                "connector_id": null,
                "created_at": 1735041372,
                "currency": "USD",
                "customer_email": "f6cd0753a831880835a0e0d23136ef66318d8303605bd37fa80b4ecf248bd303",
                "customer_id": "StripeCustomer",
                "description": "Its my first payment request",
                "feature_metadata": {
                    "redirect_response": null,
                    "search_tags": [
                        "1259195bb05bb44ea78ab60b26a54065183f91b3c5b3c2c074cadcf521305a79",
                        "9bff7b14d0a57a00d98f5eb742418a50b3bdfe5993f16b4219bfa31989bdf80f"
                    ]
                },
                "headers": {},
                "merchant_id": "merchant_1735041293",
                "merchant_order_reference_id": null,
                "metadata": "{\"data2\":\"camel\",\"login_date\":\"2019-09-10T10:11:12Z\",\"new_customer\":\"true\"}",
                "modified_at": 1735041373,
                "off_session": null,
                "organization_id": "org_jpjr9NADZXjHCMa592aw",
                "payment_confirm_source": null,
                "payment_id": "pay_5cvsiaRXZRzRg6VNvJ5e",
                "profile_id": "pro_PDu2uP7icn3iWcB7WEUI",
                "return_url": "https://google.com/",
                "setup_future_usage": null,
                "shipping_details": null,
                "source_type": "kafka",
                "statement_descriptor_name": "joseph",
                "statement_descriptor_suffix": "JS",
                "status": "succeeded",
                "tenant_id": "public",
                "timestamp": "2024-12-24T11:56:12Z"
            }
        ],
        "status": "Success"
    },
    {
        "count": 0,
        "index": "refunds",
        "hits": [],
        "status": "Success"
    },
    {
        "count": 0,
        "index": "disputes",
        "hits": [],
        "status": "Failure"
    },
    {
        "count": 0,
        "index": "sessionizer_payment_attempts",
        "hits": [],
        "status": "Failure"
    },
    {
        "count": 0,
        "index": "sessionizer_payment_intents",
        "hits": [],
        "status": "Failure"
    },
    {
        "count": 0,
        "index": "sessionizer_refunds",
        "hits": [],
        "status": "Failure"
    },
    {
        "count": 0,
        "index": "sessionizer_disputes",
        "hits": [],
        "status": "Failure"
    }
]
  • customer_id filter getting applied when building the opensearch query:
Index: PaymentIntents
Payload: {
  "query": {
    "bool": {
      "filter": [
        {
          "multi_match": {
            "type": "phrase",
            "query": "usd",
            "lenient": true
          }
        },
        {
          "terms": {
            "customer_id.keyword": [
              "StripeCustomer"
            ]
          }
        }
      ],
      "must": [
        {
          "bool": {
            "must": [
              {
                "bool": {
                  "should": [
                    {
                      "bool": {
                        "must": [
                          {
                            "term": {
                              "organization_id.keyword": {
                                "value": "org_jpjr9NADZXjHCMa592aw"
                              }
                            }
                          }
                        ]
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              }
            ]
          }
        }
      ]
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}

Checklist

  • I formatted the code cargo +nightly fmt --all
  • I addressed lints thrown by cargo clippy
  • I reviewed the submitted code
  • I added unit tests for my changes where possible

@tsdk02 tsdk02 requested a review from a team as a code owner January 20, 2025 10:18
@semanticdiff-com
Copy link

semanticdiff-com bot commented Jan 20, 2025

Review changes with  SemanticDiff

Changed Files
File Status
  crates/analytics/src/search.rs  69% smaller
  crates/analytics/src/opensearch.rs  16% smaller
  crates/api_models/src/analytics/search.rs  0% smaller

@tsdk02 tsdk02 self-assigned this Jan 20, 2025
@tsdk02 tsdk02 added C-feature Category: Feature request or enhancement A-Analytics labels Jan 20, 2025
@tsdk02 tsdk02 changed the title feat(opensearch): add amount as filter and handle name for different indexes feat(opensearch): add amount and customer_id as filters and handle name for different indexes Feb 5, 2025
@tsdk02 tsdk02 removed the request for review from ThisIsMani February 6, 2025 05:08
@likhinbopanna likhinbopanna added this pull request to the merge queue Feb 7, 2025
Merged via the queue into main with commit df328c5 Feb 7, 2025
17 of 20 checks passed
@likhinbopanna likhinbopanna deleted the amount-filter-global-search branch February 7, 2025 13:14
@juspay juspay deleted a comment from Bhagyabannur Feb 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-Analytics C-feature Category: Feature request or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(opensearch): add amount and customer_id as filters and handle name for different indexes

4 participants