Elasticsearch version (bin/elasticsearch --version): 7.16.2

Plugins installed: []

JVM version (java -version): OpenJDK Runtime Environment Temurin-17.0.1+12

OS version (uname -a if on a Unix-like system): MacOS + Docker

Description of the problem including expected versus actual behavior:

The http response of the create snapshot API does not contain the X-Elastic-Product header if wait_for_completion=true. This breaks product check in the client libraries (this was reported in elastic/elasticsearch-java#74)

Debugging the issue, I found that the header is not added because the ThreadContext's thread local in DefaultRestChannel is different from the one intialized in RestController.dispatchRequest, which contains X-Elastic-Product as a response header.

I also found that when wait_for_completion=true, the action listener is called from a snapshot thread, which may explain the issue.

Steps to reproduce:

1. Start Elasticsearch:

   docker run -t -p 9200:9200 \
     -e "discovery.type=single-node" -e "xpack.security.enabled=false" -e "path.repo=/tmp" \
     docker.elastic.co/elasticsearch/elasticsearch:7.16.2
   

2. Create a snapshot repository:

   curl -v -XPUT "http://localhost:9200/_snapshot/test" -H 'Content-Type: application/json' \
       -d '{"type":"fs","settings":{"location":"/tmp/backup"}}'
   

3. Create a snapshot, not waiting for completion. Response contains X-Elastic-Product

   curl -v -XPUT "http://localhost:9200/_snapshot/test/1
   

4. Create a snapshot, waiting for completion. Response does not contain X-Elastic-Product

   curl -v -XPUT "http://localhost:9200/_snapshot/test/2?wait_for_completion=true"