This repository was archived by the owner on Sep 17, 2025. It is now read-only.
This repository was archived by the owner on Sep 17, 2025. It is now read-only.
Handle exceptions from Exporters #270
Description
Currently if there's an exception while emitting spans, BackgroundThreadTransport will stop working.
Exception in thread opencensus.trace.Worker:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/app/service/image.binary.runfiles/pypi__opencensus_0_1_4/opencensus/trace/exporters/transports/background_thread.py", line 104, in _thread_main
self.exporter.emit(span_datas)
File "/app/service/image.binary.runfiles/pypi__opencensus_0_1_4/opencensus/trace/exporters/stackdriver_exporter.py", line 149, in emit
self.client.batch_write_spans(name, stackdriver_spans)
File "/app/service/image.binary.runfiles/pypi__google_cloud_trace_0_19_0/google/cloud/trace/client.py", line 104, in batch_write_spans
timeout=timeout)
File "/app/service/image.binary.runfiles/pypi__google_cloud_trace_0_19_0/google/cloud/trace/_gapic.py", line 95, in batch_write_spans
timeout=timeout)
File "/app/service/image.binary.runfiles/pypi__google_cloud_trace_0_19_0/google/cloud/trace_v2/gapic/trace_service_client.py", line 200, in batch_write_spans
request, retry=retry, timeout=timeout, metadata=metadata)
File "/app/service/image.binary.runfiles/pypi__google_api_core_1_2_1/google/api_core/gapic_v1/method.py", line 139, in __call__
return wrapped_func(*args, **kwargs)
File "/app/service/image.binary.runfiles/pypi__google_api_core_1_2_1/google/api_core/grpc_helpers.py", line 56, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "/app/service/image.binary.runfiles/pypi__six_1_11_0/six.py", line 737, in raise_from
raise value
ServiceUnavailable: 503 Connect Failed
Exception in thread opencensus.trace.Worker:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/app/service/image.binary.runfiles/pypi__opencensus_0_1_5/opencensus/trace/exporters/transports/background_thread.py", line 104, in _thread_main
self.exporter.emit(span_datas)
File "/app/service/image.binary.runfiles/pypi__opencensus_0_1_5/opencensus/trace/exporters/stackdriver_exporter.py", line 152, in emit
self.client.batch_write_spans(name, stackdriver_spans)
File "/app/service/image.binary.runfiles/pypi__google_cloud_trace_0_19_0/google/cloud/trace/client.py", line 104, in batch_write_spans
timeout=timeout)
File "/app/service/image.binary.runfiles/pypi__google_cloud_trace_0_19_0/google/cloud/trace/_gapic.py", line 96, in batch_write_spans
timeout=timeout)
File "/app/service/image.binary.runfiles/pypi__google_cloud_trace_0_19_0/google/cloud/trace_v2/gapic/trace_service_client.py", line 200, in batch_write_spans
request, retry=retry, timeout=timeout, metadata=metadata)
File "/app/service/image.binary.runfiles/pypi__google_api_core_1_2_1/google/api_core/gapic_v1/method.py", line 139, in __call__
return wrapped_func(*args, **kwargs)
File "/app/service/image.binary.runfiles/pypi__google_api_core_1_2_1/google/api_core/grpc_helpers.py", line 56, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "/app/service/image.binary.runfiles/pypi__six_1_11_0/six.py", line 737, in raise_from
raise value
ResourceExhausted: 429 Insufficient tokens for quota 'cloudtrace.googleapis.com/write_requests' and limit 'WriteRequestsPerMinutePerProject' of service 'cloudtrace.googleapis.com' for consumer 'project_number:<project id>'.
Should it be up to the exporters themselves to handle exceptions? There could be some weird interactions in that case, for example, if the exporter sleeps to retry when using SyncTransport.