Skip to content

Generate and send b3 tracing headers per CLI command #3309

@Samze

Description

@Samze

What's the user value of this feature request?
A single CLI command may result in multiple CAPI API requests. A unique id (vcap-request-id) is generated by go-router per api request that allows tracing issues through the system. Additionally, when enabled go-router will generate b3-trace-headers per request.

However, if you want to trace ALL requests for a particular CF CLI command you cannot, since there is no trace id that is per command and not per API call. This makes it hard to debug issues with the CLI, or to understand all the requests made for a single command (e.g. cf push).

The CLI should generate a unique ID per command and utilise the distributed b3 trace headers, see here.

This means if there is an issue, a developer can run CF_TRACE=true and see the generated b3 trace headers which can then be correlated to logs in the CF system by opeators.

Headers to set per command:

X-B3-Traceid - 128-bit in length and indicates the overall ID of the trace. Every span in a trace shares this ID. The X-B3-TraceId header is encoded as 32 lower-hex characters.

Headers to set per api call

X-B3-Spanid - 64-bit in length and indicates the position of the current operation in the trace tree. The X-B3-SpanId header is encoded as 16 lower-hex characters.

These then will be logged in different CF components, including CAPI, e.g. capi log

{
  "timestamp": "2024-11-18T21:00:58.094638225Z",
  "message": "Completed 200 vcap-request-id: 424e6816-392a-4c68-5531-cfccd8508bad::32c7ffa4-30f3-490e-8c89-96722c952549",
  "log_level": "info",
  "source": "cc.api",
  "data": {
    "request_guid": "424e6816-392a-4c68-5531-cfccd8508bad::32c7ffa4-30f3-490e-8c89-96722c952549",
    "user_guid": "8f6fb41c-b334-45e8-ba25-ba3b909bbb70",
    "b3_trace_id": "a67d95354cc07e8a87d5706c44ff5a6e",
    "b3_span_id": "4b24d3227b42b0cb666995afcef772e6",
    "status_code": 200
  },
  "thread_id": 72620,
  "fiber_id": 72640,
  "process_id": 7,
  "file": "/var/vcap/data/packages/cloud_controller_ng/19f5e6515e03b9d0de46e7ce1e34451881bc74e6/cloud_controller_ng/lib/cloud_controller/logs/request_logs.rb",
  "lineno": 32,
  "method": "complete_request"
}

It should also be possible to explicitly set the trace header.

We should introduce the following new environment variable.

export CF_B3_TRACE_ID=<my_trace_id>

In addition to this, if someone explicitly sets the header via -H using a cf curl that should take priority.

So the order of priority of this header is:

  1. cf curl -H B3_TRACE_ID=xxx
  2. CF_B3_TRACE_ID environment variable
  3. Auto generated

Who is the functionality for?
Operators debugging CF issues.

Who else is affected by the change?
Currently b3 headers are generated by go-router, this would mean that CLI requests no longer contain unique b3 headers per request but rather per command.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions