Skip to content

IMPORT PGDUMP cannot import from Azure Blob Storage in Government #47163

@nlowe

Description

@nlowe

Describe the problem

It would appear that the Account Base URL for Azure Storage is hard-coded to the Azure Public Cloud end-point:

u, err := url.Parse(fmt.Sprintf("https://%s.blob.core.windows.net", conf.AccountName))

(more recently:

u, err := url.Parse(fmt.Sprintf("https://%s.blob.core.windows.net", conf.AccountName))
)).

This prevents consumers with blob storage in Azure Government Cloud from restoring backups from their blob storage accounts.

To Reproduce

NOTE: All Azure Services in the following steps were created in the Azure US Government Cloud

  1. Setup a Secure CockroachDB Cluster in Azure AKS in following https://github.com/cockroachdb/cockroach/tree/master/cloud/kubernetes

  2. Create an Azure Storage Blob Account

  3. Create an Azure Storage Blob Container in the account created in step 2

  4. Upload backup.sql taken from cockroach dump to the storage container

  5. Create the database via cockroach sql:

     CREATE DATABASE test;
  6. Attempt to restore the database:

    IMPORT PGDUMP 'azure://mycontainer/test.sql?AZURE_ACCOUNT_NAME=myaccount&AZURE_ACCOUNT_KEY=***' WITH max_row_size='100MB';
    
    Error: pq: failed to create azure reader: -> github.com/cockroachdb/cockroach/vendor/github.com/Azure/azure-pipeline-go/pipeline.newDefaultHTTPClientFactory.func1.1, /go/src/github.com/cockroachdb/cockroach/vendor/github.com/Azure/azure-pipeline-go/pipeline/core.go:234
    HTTP request failed
    
    Get https://myaccount.blob.core.windows.net/mycontainer/test.sql?timeout=61: dial tcp: lookup myaccount.blob.core.windows.net on 192.168.0.10:53: no such host
    
    Failed running "sql"

Expected behavior

A user is either able to specify the full path to their storage account or force CockroachDB to look for storage accounts in different azure environments like US Government Cloud.

Additional data / screenshots

I also tried specifying the full account URL:

IMPORT PGDUMP 'azure://myaccount.blob.core.usgovcloudapi.net/mycontainer/test.sql?AZURE_ACCOUNT_NAME=myaccount&AZURE_ACCOUNT_KEY=***' WITH max_row_size='100MB';

but that clearly wasn't interpreted correctly:

Get https://myaccount.blob.core.windows.net/mycontainer.blob.core.usgovcloudapi.net/test.sql?timeout=61: dial tcp: lookup myaccount.blob.core.windows.net on 192.168.0.10:53: no such host

After finding this Stack Overflow Post about using the Azure GO SDKs with Azure Government, I even tried setting AZURE_ENVIRONMENT to AZUREUSGOVERNMENTCLOUD on all of my nodes including the client I was using to execute the IMPORT PGDUMP. This had no effect.

Environment:

  • CockroachDB version: v2.1.0
  • Server OS: Official Docker Containers in Kubernetes 1.16
  • Client app: cockroach sql

Additional context

Another method like HTTP or Node Local files will have to be used for backup/restore.

Jira issue: CRDB-5033

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions