Parent ticket: #101016

Summary

Background: #109276 (comment)

We should consider a "hybrid" approach in which an "update" means we write a document to the current write index (which we can assume to have the most recent mappings) and we delete the document from the index it was previously in. That would be almost like a reindex-on-write approach.

Benefits:

• In Observability old alerts can be "recovered" in a rule executor, which leads to their timestamps and many other properties being updated. In this case, it probably doesn't make a lot of sense to continue keeping them in the old indices.
• A recovered alert would move "closer" in the index (e.g. potentially move from the warm to the hot tier).
• If we update all types of alerts this way, it will allow us to only maintain compatibility of old indices with read operations, but not write/update operations. Which would simplify our lives when dealing with [RAC][Rule Registry] Write an RFC for index bootstrapping and backwards compatibility #110800 and [RAC][Rule Registry] Applying aliases/runtime fields to old indices for backwards compatibility #110808.

Risks/cons:

• We don't have a transaction to make this operation atomic, so we might have intermittently missing or duplicate alert docs.
• In Security we don't update @timestamp and other fields except the status, and the benefits are not that obvious for Security.