Skip to content
This repository was archived by the owner on Feb 26, 2026. It is now read-only.
This repository was archived by the owner on Feb 26, 2026. It is now read-only.

If VM metadata fetch fails in google.cloud.logging.Client.setup_logging(), all logging starts to fail #710

Closed
#718
Closed
If VM metadata fetch fails in google.cloud.logging.Client.setup_logging(), all logging starts to fail#710
#718
Assignees
daniel-sanche
Labels
api: loggingIssues related to the googleapis/python-logging API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
@stewartmiles

Description

@stewartmiles
stewartmiles
opened on Jan 20, 2023

Environment details

  • OS type and version: Google Cloud Container Optimized OS (gke-12314-gke401-cos-93-16623-295-14-v221129-c-pre)
  • Python version: 3.8.10
  • pip version: 22.3.1
  • google-cloud-logging version: 3.4.0

Steps to reproduce

  1. Hack retrieve_metadata_server() so that it returns test when provided metadata_key in the set {'instance/id', 'instance/zone'} and None for all other values. This can happen if a metadata server request fails.
  2. Try to log something and see unhandled exceptions from the logging thread.

Code example

test.py

import logging

import google.cloud.logging
import google.cloud.logging_v2.handlers._monitored_resources


# Fake implementation of
# google.cloud.logging_v2._helpers.retrieve_metadata_server()
# that simulates a metadata server error for all keys except
# GCE instance ID and zone.
def _fake_retrieve_metadata_server(metadata_key, timeout=5):
  if metadata_key in {'instance/id', 'instance/zone'}:
    return 'test'
  return None

# Patch the reference to
# google.cloud.logging_v2._helpers.retrieve_metadata_server
# imported in google.cloud.logging_v2.handlers._monitored_resources).
(google.cloud.logging_v2.handlers._monitored_resources.
 retrieve_metadata_server) = _fake_retrieve_metadata_server

client = google.cloud.logging.Client()
client.setup_logging()
logging.info('hello world')

Run python test.py

Stack trace

Program shutting down, attempting to send 1 queued log entries to Cloud Logging...
Waiting up to 5 seconds.
Failed to submit 1 logs.
Traceback (most recent call last):
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 550, in _ConvertFieldValuePair
    self._ConvertMapFieldValue(value, message, field)
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 697, in _ConvertMapFieldValue
    getattr(message, field.name)[key_value] = _ConvertScalarFieldValue(
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 733, in _ConvertScalarFieldValue
    if _UNPAIRED_SURROGATE_PATTERN.search(value):
TypeError: expected string or bytes-like object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 580, in _ConvertFieldValuePair
    self.ConvertMessage(value, sub_message)
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 476, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 594, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse labels field: expected string or bytes-like object.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/cloud/logging_v2/handlers/transports/background_thread.py", line 115, in _safely_commit_batch
    batch.commit()
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/cloud/logging_v2/logger.py", line 467, in commit
    client.logging_api.write_entries(
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/cloud/logging_v2/_gapic.py", line 154, in write_entries
    log_entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/cloud/logging_v2/_gapic.py", line 154, in <listcomp>
    log_entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/cloud/logging_v2/_gapic.py", line 564, in _log_entry_mapping_to_pb
    ParseDict(mapping, entry_pb)
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 445, in ParseDict
    parser.ConvertMessage(js_dict, message)
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 476, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 588, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse resource field: Failed to parse labels field: expected string or bytes-like object..
Sent all pending logs.

Metadata

Metadata

Assignees

Labels

api: loggingIssues related to the googleapis/python-logging API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions