Elasticsearch version (7.7):

JVM version (jdk-11):

Description of the problem including expected versus actual behavior:

1. I was seeing an issue where the JVM usage of the ES process went up very high to 97% and was analyzing the heap dump for it. In this case, one shard of an index end up consuming around 25GB of JVM out of 32GB. The index was receiving very high ingestion workload. I was surprised to see that and was trying to understand why indexingBuffer based throttling didn't kicked in. I can see from the dump, that the indexingBuffer limit is breached for one ES shard (or Lucene index) and IndexingMemoryController detects that and activates the throttling mode for the shard. But I was expecting that with throttling mode activated the InternalEngine will start throttling any new indexing requests and also initiates the flush process to relieve the memory pressure. However, while looking into the code it looks like InternalEngine doesn't throttle the new indexing requests if it's not because of recovery. It looks to me even though the indexingBuffer limit is breached by one of the Lucene index, it can still keep consuming the memory by the new indexing requests and increase the overall memory usage by the Lucene index or ES shard. If the ingestion workload is very high this can consume memory very fast.

Ref: bd1c30f#diff-74a25d57bbb52202a04fe43d8632172eR547

It looks like with this commit (#22721) the check was reversed to apply throttling only in recovery scenarios.

Please let me know if my understanding is correct. I can open a PR with the fix.