Overview

There are few problems with the initial approach of parse command implementation from #411, it brings some limitations described in #359.

One of the root cause is that PPL cannot determine whether to use NamedExpression or ParseExpression when resolving identifiers. For example:

source=index | parse message "(?<firstChar>\w).*" | fields firstChar

In this query when resolving firstChar, PPL should refer to ParseExpression firstChar and derive it from field message.

source=index | parse message "(?<firstChar>\w).*" | stats count() by firstChar | fields count(), firstChar

In this query behavior should be the same when resolving firstChar in stats, but in fields command, firstChar should come from NamedExpression because message no longer exists.

To solve this issue, initial implementation always prioritizes to use ParseExpression instead of NamedExpression if both exists. If the raw text field of parse expression is missing, then it will read from tuple values. This brings the restrictions of not able to rewrite parsed fields and etc. Related code:

Ideas for a better implementation:

1. If OpenSearch supports creating fields at runtime, parse command can be pushed down to that.
2. Do not convert parse command as parse expressions in analyzer, do it here instead?
   

   sql/opensearch/src/main/java/org/opensearch/sql/opensearch/storage/OpenSearchIndexScan.java

   Lines 76 to 77 in fe20fb1

   	this.request = new OpenSearchQueryRequest(indexName,
   	settings.getSettingValue(Settings.Key.QUERY_SIZE_LIMIT), exprValueFactory);