Elasticsearch Version

8.16.1

Installed Plugins

Unknown

Java Version

ECP

OS Version

ECP

Problem Description

It is possible that the same Watchers were getting triggered on multiple nodes simultaneously. This results in one execution succeeding while the other fails with a version_conflict_engine_exception error on the .watches index.

Steps to Reproduce

Create a cluster with a large (6k+) number of watches, place cluster under load, add replicas for .watches shards, wait for .watches shards to re-locate a few times, observe concurrent write errors as watch executes on multiple nodes and fails to write state due to conflicts

Logs (if relevant)

See mentioned issues