Skip to content

Fix 'autoGeneratedTimestamp should not be set externally' error when retrying IndexRequest#86184

Merged
martijnvg merged 9 commits intoelastic:masterfrom
martijnvg:fix_autoGeneratedTimestamp_error_when_retryin_index_request
Apr 28, 2022
Merged

Fix 'autoGeneratedTimestamp should not be set externally' error when retrying IndexRequest#86184
martijnvg merged 9 commits intoelastic:masterfrom
martijnvg:fix_autoGeneratedTimestamp_error_when_retryin_index_request

Conversation

@martijnvg
Copy link
Copy Markdown
Member

@martijnvg martijnvg commented Apr 26, 2022

We use BulkProcessor to index various features internally (ilm history, watcher history and more).
When an index request is retried then resending that same request can cause the following error:

java.lang.IllegalArgumentException: autoGeneratedTimestamp should not be set externally
	at org.elasticsearch.action.bulk.TransportBulkAction.doInternalExecute(TransportBulkAction.java:223) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.bulk.TransportBulkAction$1.doRun(TransportBulkAction.java:192) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:200) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:91) [elasticsearch-7.16.1.jar:7.16.1]

This happens when index requests fail because of a retryable error (e.g. EsRejectedExecutionException).
Prior to this change, the BulkProcessor (via Retry) then reuses the same instance.
With this change, the autoGeneratedTimestamp field is unset via a reset method added to IndexRequest class,
which should avoid the IAE.

Closes #83927

@elasticmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Hi @martijnvg, I've created a changelog YAML for you.

…retrying IndexRequest

We use `BulkProcessor` to index various features internally (ilm history, watcher history and more).
When an index request is retried then resending that same request can cause the following error:

```
java.lang.IllegalArgumentException: autoGeneratedTimestamp should not be set externally
	at org.elasticsearch.action.bulk.TransportBulkAction.doInternalExecute(TransportBulkAction.java:223) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.bulk.TransportBulkAction$1.doRun(TransportBulkAction.java:192) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:200) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:91) [elasticsearch-7.16.1.jar:7.16.1]
```

This happens when index requests fail because of a retryable error (e.g. EsRejectedExecutionException).
Prior to this change, the `BulkProcessor` (via `Retry`) then reuses the same instance.
With this change, a copy is made of the original `IndexRequest`, so that the retry is performed with an unset
`autoGeneratedTimestamp` field.

Closes elastic#83927
@martijnvg martijnvg force-pushed the fix_autoGeneratedTimestamp_error_when_retryin_index_request branch from 1fa52b5 to fc170e2 Compare April 26, 2022 13:03
@dakrone
Copy link
Copy Markdown
Member

dakrone commented Apr 26, 2022

@martijnvg what's your thought on this solution versus doing something like adding a .reset() method on the IndexRequest that allows resetting that internal field so it can be retried? Do you have a preference?

@martijnvg
Copy link
Copy Markdown
Member Author

@dakrone I think I'm in favour of that instead of creating a new instance based on the original. The copying is error prone, especially if new fields are added in the future to IndexRequest. I will add the reset method and remove the IndexRequest best effort cloning.

Copy link
Copy Markdown
Member

@dakrone dakrone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM assuming CI is happy. Do we have a way to add a test for this that doesn't bend over backwards in complexity?

@martijnvg
Copy link
Copy Markdown
Member Author

Do we have a way to add a test for this that doesn't bend over backwards in complexity?

I don't think so? This doesn't change behaviour or formats. The logic here is tested via BulkProcessorRetryIT, AsyncBulkByScrollActionTests and RetryTests (these test failed when making a mistake with the previous approach).
So I think we're good here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

>bug Team:Data Management (obsolete) DO NOT USE. This team no longer exists. v8.3.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Our internal usage of BulkProcessor can fail indexing during retry

4 participants