Skip to content

[BUG] Circuit breaker getting triggered when multiple PPL queries are fired in parallel. #4771

@RyanL1997

Description

@RyanL1997

Description

This issue transfers a experiment @paulstn and I did in OSD's new discover experience. When loading into the discover page at OSD, it fires multiple queries against each index field to calculate the presentation percentage.

Example queries:

# query 1
source = demo-ai-logs-otel-v1* 
  | WHERE `@timestamp` >= '2025-07-29 22:53:20.604' AND `@timestamp` <= '2025-10-27 22:53:20.604' 
  | where isnotnull(`attributes.State`) 
  | stats count() as field_count, distinct_count(`attributes.State`) as distinct_count

# query 2
source = demo-ai-logs-otel-v1* 
  | WHERE `@timestamp` >= '2025-07-29 22:53:20.604' AND `@timestamp` <= '2025-10-27 22:53:20.604' 
  | where isnotnull(`attributes.address`) | stats count() as field_count, distinct_count(`attributes.address`) as distinct_count
...

Example errors:

...
{
  "text": "{\"statusCode\":400,\"error\":\"Bad Request\",\"message\":\"{\\n  \\\"error\\\": {\\n    \\\"reason\\\": \\\"Error occurred in OpenSearch   engine: all shards failed\\\",\\n    
  \\\"details\\\": \\\"Shard[0]: [demo-ai-logs-otel-v1-00001/oIkyldJtSHm6Cwgnj-aqxA] QueryShardException[failed to create query: 
  Failed to compile inline script   [rO0ABXNyADRvcmcub3BlbnNlYXJjaC5zcWwuZXhwcmVzc2lvbi5mdW5jdGlvbi5GdW5jdGlvbkRTTCQyPc501CEBPWwCAAVMAA12YWwkYXJn
  ...
  ...
  dGltZS5TZXKVXYS6GyJIsgwAAHhwdwYHAANVVEN4c3EAfgBNdw0CAAAAAGj/9+AtYkYHeH5xAH4AG3QAB0JPT0xFQU4=] using lang
  [opensearch_query_expression]]; 
  nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, 
  or scripts with parameters instead; this limit can be changed by the [script.context.filter.max_compilations_rate] setting];\\\\n\\\\n
  For more details, please send request for Json format to see the raw response from OpenSearch engine.\\\",\\n    
  \\\"type\\\": \\\"SearchPhaseExecutionException\\\"\\n  },\\n  \\\"status\\\": 400\\n}\"}"
  },
  "redirectURL": "",
  "headersSize": 800,
  "bodySize": 3641
},
...

Env

  • OS: 3.3

Screenshots

Image

Potential Work Around

Instead of having multiple queries fired front frontend, do a single query.

Exit Criteria:

As SQL plugin, here are couple action items:

  • Investigate the current default limitation of script.context.filter.max_compilations_rate
  • Implement the fix/enhancement to address the above issue

Metadata

Metadata

Assignees

No one assigned

    Labels

    PPLPiped processing languagebugSomething isn't workingengine v2Issues related to v2 query engine only.error-experienceIssues related to how we handle failure cases in the plugin.performanceMake it fast!

    Type

    No type

    Projects

    Status

    Done

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions