Add support for HTTP Proxies for the GCS repository by arteam · Pull Request #82737 · elastic/elasticsearch

arteam · 2022-01-18T15:18:03Z

The change adds 3 new client properties for the GCS repository:

gcs.client.default.proxy.type
gcs.client.default.proxy.host
gcs.client.default.proxy.port

They allow to configure a java.net.Proxy for the GCS SDK to use when communicating with the GCS API.

Resolves #82444

elasticmachine · 2022-01-18T15:18:07Z

Pinging @elastic/es-distributed (Team:Distributed)

The change adds 3 new client properties for the GCS repository: * gcs.client.default.proxy.type * gcs.client.default.proxy.host * gcs.client.default.proxy.port They allow to configure a [java.net.Proxy](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/Proxy.html) for the GCS SDK to use when communicating with the GCS API. Resolves elastic#82444

docs/reference/snapshot-restore/repository-gcs.asciidoc

arteam · 2022-01-20T09:33:27Z

@elasticmachine update branch

tlrx

LGTM

arteam · 2022-01-20T14:47:10Z

Thanks Tanguy!

* Add support for HTTP Proxies for the GCS repository The change adds 3 new client properties for the GCS repository: * gcs.client.default.proxy.type * gcs.client.default.proxy.host * gcs.client.default.proxy.port They allow to configure a [java.net.Proxy](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/Proxy.html) for the GCS SDK to use when communicating with the GCS API. Resolves elastic#82444

elasticsearchmachine · 2022-01-20T14:48:01Z

💚 Backport successful

Status	Branch	Result
✅	8.0

The change adds 3 new client properties for the GCS repository: * gcs.client.default.proxy.type * gcs.client.default.proxy.host * gcs.client.default.proxy.port They allow to configure a [java.net.Proxy](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/Proxy.html) for the GCS SDK to use when communicating with the GCS API. Resolves #82444

sangeethdba · 2022-02-12T06:26:48Z

does these settings work with elastic 6.8.2 or 7.x

sangeethdba · 2022-02-12T06:27:35Z

path: /_snapshot/backup, params: {pretty=, repository=backup}
org.elasticsearch.repositories.RepositoryException: [backup] cannot create blob store
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:328) ~[elasticsearch-6.8.2.jar:6.8.2]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:625) ~[elasticsearch-6.8.2.jar:6.8.2]
at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$2(RepositoriesService.java:230) ~[elasticsearch-6.8.2.jar:6.8.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.8.2.jar:6.8.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
Caused by: org.elasticsearch.common.blobstore.BlobStoreException: Unable to check if bucket [com-dbre-backup-np] exists
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:109) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.(GoogleCloudStorageBlobStore.java:75) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:137) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:47) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:324) ~[elasticsearch-6.8.2.jar:6.8.2]
... 6 more
Caused by: com.google.cloud.storage.StorageException: Error getting access token for service account:
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:226) ~[?:?]
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:403) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:196) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:193) ~[?:?]
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89) ~[?:?]
at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:192) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$doesBucketExist$0(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_322]
at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.(GoogleCloudStorageBlobStore.java:75) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:137) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:47) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:324) ~[elasticsearch-6.8.2.jar:6.8.2]
... 6 more
Caused by: java.io.IOException: Error getting access token for service account:
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:395) ~[?:?]
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:179) ~[?:?]
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:165) ~[?:?]
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96) ~[?:?]
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159) ~[?:?]
at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109) ~[?:?]
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469) ~[?:?]
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:400) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:196) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:193) ~[?:?]
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89) ~[?:?]
at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:192) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$doesBucketExist$0(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_322]
at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.(GoogleCloudStorageBlobStore.java:75) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:137) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:47) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:324) ~[elasticsearch-6.8.2.jar:6.8.2]
... 6 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_322]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_322]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_322]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_322]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_322]
at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_322]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:288) ~[?:?]
at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:203) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:189) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1342) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1317) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264) ~[?:?]
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77) ~[?:?]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981) ~[?:?]
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:393) ~[?:?]
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:179) ~[?:?]
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:165) ~[?:?]
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96) ~[?:?]
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159) ~[?:?]
at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109) ~[?:?]
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469) ~[?:?]
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:400) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:196) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:193) ~[?:?]
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89) ~[?:?]
at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:192) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$doesBucketExist$0(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_322]
at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.(GoogleCloudStorageBlobStore.java:75) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:137) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:47) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:324) ~[elasticsearch-6.8.2.jar:6.8.2]

sangeethdba · 2022-02-12T06:28:33Z

@tlrx @arteam

Re-adds HTTP proxy settings for the GCS repository type. These settings were added with #82737. The docs were accidentally removed as part of #82996.

Re-adds HTTP proxy settings for the GCS repository type. These settings were added with #82737. The docs were accidentally removed as part of #82996. (cherry picked from commit e8b34c7)

We've had a few users ask about support for HTTP proxy connections to GCS snapshot repositories in 7.x and 6.x. However, client settings for those connections are only supported in 8.0+. This adds a note to the 7.17 and 6.8 docs to nudge users to upgrade. Relates to #82737 Closes #83959

) We've had a few users ask about support for HTTP proxy connections to GCS snapshot repositories in 7.x and 6.x. However, client settings for those connections are only supported in 8.0+. This adds a note to the 7.17 and 6.8 docs to nudge users to upgrade. Relates to #82737 Closes #83959 (cherry picked from commit 99de635)

…ijun/elasticsearch into fix-none-tsdb-index-dimension-tests * 'fix-none-tsdb-index-dimension-tests' of github.com:weizijun/elasticsearch: (37 commits) [docs] Mention JDK 17 in the Contributing docs (elastic#84018) Fix GeoIpDownloader startup during rolling upgrade (elastic#84000) Script: Fields API for Dense Vector (elastic#83550) Move InferenceConfigUpdate under VersionedNamedWriteable (elastic#84022) [ML] Fix license feature test cleanup (elastic#84020) Replace deprecated api in artifact transforms (elastic#84015) QL: Add leniency option to SQL CLI (elastic#83795) [Stack Monitoring] add kibana_stats version alias to -mb template (elastic#83930) Optimize spliterator for ImmutableOpenMap (elastic#83899) Feature usage actions for archive (elastic#83931) Use latch to speedup multi feature migration test (elastic#84007) Make action names available in NodeClient (elastic#83919) [DOCS] Re-add HTTP proxy setings from elastic#82737 (elastic#84001) Add CI matrix configuration for snapshot BWC versions (elastic#83990) Update YAML Rest tests to check for product header on all responses (elastic#83290) TSDB: Add time series aggs cancellation (elastic#83492) [DOCS] Fix percolate query headings (elastic#83988) [DOCS] Move tip for percolate query example (elastic#83972) Simplify LocalExporter cleaner function to fix failing tests (elastic#83812) [GCE Discovery] Correcly handle large zones with 500 or more instances (elastic#83785) ...

They were added in elastic#82737, but not exposed via the GCSPlugin to end users.

They were added in #82737, but not exposed via the GCSPlugin to end users.

They were added in elastic#82737, but not exposed via the GCSPlugin to end users.

They were added in #82737, but not exposed via the GCSPlugin to end users.

arteam added >enhancement :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.0.0 auto-backport Automatically create backport pull requests when merged labels Jan 18, 2022

elasticmachine added the Team:Distributed Meta label for distributed team. label Jan 18, 2022

elasticsearchmachine added the v8.1.0 label Jan 18, 2022

arteam force-pushed the gcs-proxy branch from 7e59324 to 9eff0df Compare January 18, 2022 15:37

arteam requested review from original-brownbear and tlrx January 18, 2022 16:37

tlrx reviewed Jan 19, 2022

View reviewed changes

docs/reference/snapshot-restore/repository-gcs.asciidoc Outdated Show resolved Hide resolved

Remove examples of proxy parameters

b70c99b

Merge branch 'master' into gcs-proxy

9aa042e

arteam requested a review from tlrx January 20, 2022 11:36

tlrx approved these changes Jan 20, 2022

View reviewed changes

arteam merged commit 1ddaf25 into elastic:master Jan 20, 2022

arteam mentioned this pull request Jan 20, 2022

[8.0] Add support for HTTP Proxies for the GCS repository (#82737) #82865

Merged

arteam deleted the gcs-proxy branch January 20, 2022 14:53

This was referenced Feb 15, 2022

[DOCS] Note HTTP proxy settings for GCS snapshot repo are only available in 8.0+ #83959

Closed

[DOCS] Re-add HTTP proxy settings for GCS repository type #84001

Merged

jrodewig mentioned this pull request Feb 15, 2022

[DOCS] Add note for HTTP proxy connections to GCS repos #84004

Merged

arteam added a commit to arteam/elasticsearch that referenced this pull request Apr 11, 2022

Expose proxy settings for GCS repositories

6fd4f2d

They were added in elastic#82737, but not exposed via the GCSPlugin to end users.

arteam mentioned this pull request Apr 11, 2022

Expose proxy settings for GCS repositories #85785

Merged

arteam added a commit that referenced this pull request Apr 12, 2022

Expose proxy settings for GCS repositories (#85785)

026fb09

They were added in #82737, but not exposed via the GCSPlugin to end users.

arteam added a commit to arteam/elasticsearch that referenced this pull request Apr 12, 2022

Expose proxy settings for GCS repositories (elastic#85785)

f5b07d4

They were added in elastic#82737, but not exposed via the GCSPlugin to end users.

arteam added a commit to arteam/elasticsearch that referenced this pull request Apr 12, 2022

Expose proxy settings for GCS repositories (elastic#85785)

608ab65

They were added in elastic#82737, but not exposed via the GCSPlugin to end users.

arteam added a commit to arteam/elasticsearch that referenced this pull request Apr 12, 2022

Expose proxy settings for GCS repositories (elastic#85785)

b58ad0e

They were added in elastic#82737, but not exposed via the GCSPlugin to end users.

arteam added a commit that referenced this pull request Apr 12, 2022

Expose proxy settings for GCS repositories (#85785) (#85831)

364269a

They were added in #82737, but not exposed via the GCSPlugin to end users.

arteam added a commit that referenced this pull request Apr 12, 2022

Expose proxy settings for GCS repositories (#85785) (#85832)

de4adff

They were added in #82737, but not exposed via the GCSPlugin to end users.

arteam added a commit that referenced this pull request Apr 12, 2022

Expose proxy settings for GCS repositories (#85785) (#85833)

8ba1a5c

They were added in #82737, but not exposed via the GCSPlugin to end users.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for HTTP Proxies for the GCS repository#82737

Add support for HTTP Proxies for the GCS repository#82737
arteam merged 3 commits intoelastic:masterfrom
arteam:gcs-proxy

arteam commented Jan 18, 2022 •

edited

Loading

Uh oh!

elasticmachine commented Jan 18, 2022

Uh oh!

Uh oh!

arteam commented Jan 20, 2022

Uh oh!

tlrx left a comment

Uh oh!

arteam commented Jan 20, 2022 •

edited

Loading

Uh oh!

elasticsearchmachine commented Jan 20, 2022

Uh oh!

sangeethdba commented Feb 12, 2022

Uh oh!

sangeethdba commented Feb 12, 2022

Uh oh!

sangeethdba commented Feb 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

arteam commented Jan 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Jan 18, 2022

Uh oh!

Uh oh!

arteam commented Jan 20, 2022

Uh oh!

tlrx left a comment

Choose a reason for hiding this comment

Uh oh!

arteam commented Jan 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jan 20, 2022

💚 Backport successful

Uh oh!

sangeethdba commented Feb 12, 2022

Uh oh!

sangeethdba commented Feb 12, 2022

Uh oh!

sangeethdba commented Feb 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

arteam commented Jan 18, 2022 •

edited

Loading

arteam commented Jan 20, 2022 •

edited

Loading