-
Notifications
You must be signed in to change notification settings - Fork 4.1k
IMPORT PGDUMP cannot import from Azure Blob Storage in Government #47163
Description
Describe the problem
It would appear that the Account Base URL for Azure Storage is hard-coded to the Azure Public Cloud end-point:
| u, err := url.Parse(fmt.Sprintf("https://%s.blob.core.windows.net", conf.AccountName)) |
(more recently:
| u, err := url.Parse(fmt.Sprintf("https://%s.blob.core.windows.net", conf.AccountName)) |
This prevents consumers with blob storage in Azure Government Cloud from restoring backups from their blob storage accounts.
To Reproduce
NOTE: All Azure Services in the following steps were created in the Azure US Government Cloud
-
Setup a Secure CockroachDB Cluster in Azure AKS in following https://github.com/cockroachdb/cockroach/tree/master/cloud/kubernetes
-
Create an Azure Storage Blob Account
-
Create an Azure Storage Blob Container in the account created in step 2
-
Upload
backup.sqltaken fromcockroach dumpto the storage container -
Create the database via
cockroach sql:CREATE DATABASE test;
-
Attempt to restore the database:
IMPORT PGDUMP 'azure://mycontainer/test.sql?AZURE_ACCOUNT_NAME=myaccount&AZURE_ACCOUNT_KEY=***' WITH max_row_size='100MB'; Error: pq: failed to create azure reader: -> github.com/cockroachdb/cockroach/vendor/github.com/Azure/azure-pipeline-go/pipeline.newDefaultHTTPClientFactory.func1.1, /go/src/github.com/cockroachdb/cockroach/vendor/github.com/Azure/azure-pipeline-go/pipeline/core.go:234 HTTP request failed Get https://myaccount.blob.core.windows.net/mycontainer/test.sql?timeout=61: dial tcp: lookup myaccount.blob.core.windows.net on 192.168.0.10:53: no such host Failed running "sql"
Expected behavior
A user is either able to specify the full path to their storage account or force CockroachDB to look for storage accounts in different azure environments like US Government Cloud.
Additional data / screenshots
I also tried specifying the full account URL:
IMPORT PGDUMP 'azure://myaccount.blob.core.usgovcloudapi.net/mycontainer/test.sql?AZURE_ACCOUNT_NAME=myaccount&AZURE_ACCOUNT_KEY=***' WITH max_row_size='100MB';but that clearly wasn't interpreted correctly:
Get https://myaccount.blob.core.windows.net/mycontainer.blob.core.usgovcloudapi.net/test.sql?timeout=61: dial tcp: lookup myaccount.blob.core.windows.net on 192.168.0.10:53: no such hostAfter finding this Stack Overflow Post about using the Azure GO SDKs with Azure Government, I even tried setting AZURE_ENVIRONMENT to AZUREUSGOVERNMENTCLOUD on all of my nodes including the client I was using to execute the IMPORT PGDUMP. This had no effect.
Environment:
- CockroachDB version:
v2.1.0 - Server OS: Official Docker Containers in Kubernetes 1.16
- Client app:
cockroach sql
Additional context
Another method like HTTP or Node Local files will have to be used for backup/restore.
Jira issue: CRDB-5033