bug: Unable to add CA certificate for ELK plugin destination. #11178
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
Using CloudQuery sync with Azure as source plugin and Elasticsearch as destination plugin with the following yaml configuration files:
Azure:
kind: source
spec:
name: "azure"
registry: "github"
path: "cloudquery/azure"
version: "v4.3.0"
destinations: ["elasticsearch"]
spec:
subscriptions: ["00000000-0000-0000-0000-00000000"]_Elastisearch:
kind: destination
spec:
name: elasticsearch
path: cloudquery/elasticsearch
version: "v1.1.5"
write_mode: "overwrite-delete-stale"
spec:
addresses: ["https://elasticip:port"] # optional
api_key: "api-key-goes-here"
ca_cert: '${file:/path/to/certificate/file/certificate.pem}'
certificate_fingerprint: "00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00"_With this configuration files, the sync flag is giving me the following error:
_Error: failed to sync v1 source azure: failed to initialize destination plugin client for elasticsearch: destination configure: failed to call Configure: rpc error: code = Unknown desc = failed to create Elasticsearch client: error creating transport: unable to add CA certificate_
Tried with double quotes on the file variable substitution, as well as just using the path as a value for ca_certs
Expected Behavior
The sync works properly, sending the source plugin data to the destination.
CloudQuery (redacted) config
Azure:
kind: source
spec:
name: "azure"
registry: "github"
path: "cloudquery/azure"
version: "v4.3.0"
destinations: ["elasticsearch"]
spec:
subscriptions: ["00000000-0000-0000-0000-00000000"]_Elastisearch:
kind: destination
spec:
name: elasticsearch
path: cloudquery/elasticsearch
version: "v1.1.5"
write_mode: "overwrite-delete-stale"
spec:
addresses: ["https://elasticip:port"] # optional
api_key: "api-key-goes-here"
ca_cert: '${file:/path/to/certificate/file/certificate.pem}'
certificate_fingerprint: "00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00"_Steps To Reproduce
- In a linux (Ubuntu 20.04) environment
- With the config pasted in the previous step.
- Execute "./cloudquery sync azurelab.yaml elasticsearch.yaml"
- See error:
_Error: failed to sync v1 source azure: failed to initialize destination plugin client for elasticsearch: destination configure: failed to call Configure: rpc error: code = Unknown desc = failed to create Elasticsearch client: error creating transport: unable to add CA certificate_
CloudQuery (redacted) logs
2023-06-01T16:01:09Z INF Loading spec(s) args=["azurelab.yaml","elasticsearch.yaml"] module=cli
2023-06-01T16:01:10Z INF Source plugin server listening address=/tmp/cq-LdtxJGjOpLxlFodM.sock module=cli
2023-06-01T16:01:11Z INF Start sync destinations=["elasticsearch (v1.1.5)"] module=cli source="azure (v4.3.0)" sync_time=2023-06-01T16:01:11Z
2023-06-01T16:01:11Z INF Destination plugin server listening address=/tmp/cq-zikkUzZEDSuHZASc.sock module=cli
2023-06-01T16:01:12Z INF started call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-06-01T16:01:12Z grpc.time_ms=0.025 module=cli peer.address=@ protocol=grpc
2023-06-01T16:01:12Z ERR finished call grpc.code=Unknown grpc.component=server grpc.error="failed to create Elasticsearch client: error creating transport: unable to add CA certificate" grpc.method=Configure grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-06-01T16:01:12Z grpc.time_ms=0.372 module=cli peer.address=@ protocol=grpc
2023-06-01T16:01:12Z INF waiting for destination plugin to terminate module=cli
2023-06-01T16:01:12Z INF Got stop signal. Destination plugin server shutting down address=/tmp/cq-zikkUzZEDSuHZASc.sock module=cli signal=interrupt
2023-06-01T16:01:12Z INF End sync destinations=["elasticsearch (v1.1.5)"] module=cli source="azure (v4.3.0)" sync_time=2023-06-01T16:01:11Z
2023-06-01T16:01:12Z INF waiting for source plugin to terminate module=cli
2023-06-01T16:01:12Z INF Got stop signal. Source plugin server shutting down address=/tmp/cq-LdtxJGjOpLxlFodM.sock module=cli signal=interrupt
2023-06-01T16:01:12Z INF Sending sync summary to analyticsv1.cloudquery.io:443 module=cli
2023-06-01T16:01:12Z ERR exiting with error error="failed to sync v1 source azure: failed to initialize destination plugin client for elasticsearch: destination configure: failed to call Configure: rpc error: code = Unknown desc = failed to create Elasticsearch client: error creating transport: unable to add CA certificate" module=cli
CloudQuery version
2.5.3
Additional Context
No response
Pull request (optional)
- I can submit a pull request