Skip to content

[core] Implementation of the tracing package#2

Merged
palazzem merged 38 commits into
masterfrom
palazzem/trace-span-structs
Sep 15, 2016
Merged

[core] Implementation of the tracing package#2
palazzem merged 38 commits into
masterfrom
palazzem/trace-span-structs

Conversation

@palazzem

@palazzem palazzem commented Sep 9, 2016

Copy link
Copy Markdown
Contributor

Core implementation to handle spans and the tracer objects. The following is a working example of our tracer package:

package main

import (
    "fmt"
    "time"

    "golang.org/x/net/context"

    // import the Datadog tracer
    "github.com/DataDog/dd-trace-go/pkg/tracer"
)

var smallWait = time.Millisecond * 30

func main() {
    fmt.Println("Starting the fake application")
    for {
        // creation of the context object is delegated to the user
        ctx := context.Background()

        // main request
        span := tracer.NewSpan("pylons.request", "pylons", "/")
        time.Sleep(smallWait)

        ctx = tracer.ContextWithSpan(ctx, span)
        generateTemplate(ctx)

        // cache stuff
        cacheSpan := tracer.NewChildSpan("redis.command", span)
        time.Sleep(smallWait)
        cacheSpan.Finish()

        // close the parent span
        span.Finish()
    }
}

func generateTemplate(ctx context.Context) {
    parent, ok := tracer.SpanFromContext(ctx)

    // template stuff
    templateSpan := tracer.NewChildSpan("pylons.template", parent)
    defer templateSpan.Finish()
    templateSpan.Resource = "something"
    time.Sleep(smallWait)
}

What is missing?

  • tests are not pushed until this approach is accepted
  • the dispatch implementation
  • remove some parameters from the NewChildSpan() signature. We may just:
child := NewChildSpan("something", parent)
  • handling the context.Context object so that the ctx can be passed as argument between functions. A ctx must be used to retrieve a span if a parent is available.

Comment thread pkg/tracer/span.go Outdated
if !s.IsFinished() {
s.Duration = Now() - s.Start

go func() {

@clutchski clutchski Sep 9, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two issues with this:

  • spawning a goroutine per finished span tons of overhead.
  • as you commented, this could block forever, which is completely unacceptable for client side code. a channel is not the right tool for this because if a reader crashes a customer app will lock up forever.

i'd much prefer if we took this bit of code from dd-go/dogtrace. this is already running on high traffic production systems, so let's just use it instead of inventing something new. (except for the onFinish stuff, that's completely unnecessary)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK it was a proposal. I can update this code with the delivery process that we're using in the dogtrace

@palazzem palazzem Sep 10, 2016

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading some stuff around, it seems I can easily solve the point 2 (blocked go-routine) with a timeout so that if the reader crashes (the reader is the one that sends all spans to the locale/remote agent) the span is lost and the go-routine is killed. I think it makes sense because if the go-routine that sends data is dead, it's OK to lose the span because no one is capable to send it somewhere.

On the other hand, if we're using the current approach in dogtrace (without a go-routine), our code will block as you describe because it should obtain a lock and it will surely add some delay (maybe it should wait for another Span that has the lock?). Furthermore if the sender dies, the shared data structures (the children []*Span in the Span or a common global outgoingSpans []*Span) will grow without control causing a memory leak.

About the go-routine overhead, I think I can profile it because it's a built-in functionality and reading to golang.org blog posts, they were created for similar cases.

By the way I have a free weekend so I want to play a little bit with this stuff 😄
If you agree, I prefer to revert this part at the end so that I can spend some time with benchmarks and experiments.

@clutchski

clutchski commented Sep 9, 2016

Copy link
Copy Markdown
Contributor

i think we should only create spans from a tracer. and never from other spans. we've had APIs like this before and it's more confusing (when do i call span.Nest or tracer.NewSpan?)

i think something like this is simpler:

tracer := datadog.NewTracer()

// create a new root span with some info
span := tracer.NewSpan("whatever")
defer span.Finish()
span.Service = "foo"
span.Resource = "whatever"

// create a span from another
child := tracer.NewChildSpan("another", span)
defer child.Finish()

// or from a special `datadog_trace` variable in context
child2 := tracer.NewSpanFromContext("yahoo", context)
child2.Finish()

@palazzem

Copy link
Copy Markdown
Contributor Author

Yeah I agree. Having an API where developers must always use the tracer default (or custom) instance is better. I've changed the high-level API according to your suggestions.

Comment thread pkg/tracer/tracer.go
import (
"sync"

log "github.com/cihub/seelog"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a remark, but we definitely don't want to pull this dep in the final package. Also, seelog is madness, it often shows up in a top 30 cpu profile in our apps 😱

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah OK.... I've seen that in a lot of our projects. There is something else I can use?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should use a logging lib at all, or just use the primitive "log" package in Go. Do we need the user to pull another dependency for logging if he already has one?
Also we should try to keep our logging at a minimum (except if the user toggles debug mode).

@palazzem palazzem Sep 12, 2016

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, totally agree about limit our logging to warnings/errors. About the logging lib I don't know... I mean if the primitive log package provides all that we need, I think we can replace everything with that.

@palazzem palazzem added the core label Sep 10, 2016
@palazzem palazzem force-pushed the palazzem/trace-span-structs branch from ab40cc4 to c975ff8 Compare September 10, 2016 14:59
Comment thread pkg/tracer/span.go Outdated
"math/rand"
"time"

log "github.com/cihub/seelog"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LeoCavaille what do you think about that library (https://github.com/op/go-logging)? Have you ever used it?

Comment thread pkg/tracer/span.go Outdated
Start int64 `json:"start"` // span start time expressed in nanoseconds since epoch
Duration int64 `json:"duration"` // duration of the span expressed in nanoseconds
Error int32 `json:"error"` // error status of the span; 0 means no errors
Meta map[string]string `json:"meta"` // arbitrary map of metadata

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LeoCavaille may I add the omitempty for Meta and Metrics? the agent is expecting the meta field in any case even if it's nil?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if it's not in the payload, it will just set it to nil when decoding, so yea.

@palazzem

Copy link
Copy Markdown
Contributor Author

Side note: as a last step I'll implement a kind of tracer.Disable() and tracer.Enable() so that customers that have a huge instrumented codebase, can easily disable the tracer (with one line) if something bad happens.

Comment thread pkg/tracer/tracer.go
// it's better to be defensive and to produce a wrongly configured span
// that is not sent to the trace agent.
if parent == nil {
return newSpan(name, "", "", spanID, spanID, spanID, nil)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not so happy about using nil in the tracer field... we could forget that when the client handles more things. At the same time I don't know if makes sense allowing users to have more than one tracer instances.

  • Pros: I can have multiple tracers for different sections so that I can enable/disable them individually
  • Cons: have to send *tracer "around the code"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tracer could be an interfacer. CanonicalTracer could be the implementation you have now that is used by default. DevNullTracer could be an implementation that simply discards any spans given to it, and you could pass that in for the nil case. I think this should solve your problem, but at the expense of a layer of indirection in the code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I was thinking the same.... I'll leave that as is for now but I want to implement your suggestion. I'm feeling safe if nil pointers don't travel around the code 😄

@talwai talwai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reviewed this PR and contributed my humble feedback

// the child is finished but it's not recorded in
// the tracer buffer
assert.True(child.Duration > 0)
assert.Equal(len(tracer.finishedSpans), 0)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Comment thread go.rb Outdated
def go_benchmark(path)
sh "go test -run=NONE -bench=. -memprofile=mem.out #{path}"
sh "go test -run=NONE -bench=. -cpuprofile=cpu.out #{path}"
sh "go test -run=NONE -bench=. -blockprofile=block.out #{path}"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or run them all together ?
go test ... -memprofile=mem.out -cpuprofile=cpu.out -blockprofile=block.out ...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I don't have a lot of experience on that. I split them because I've found in The Go programming language book such sentence:

Gathering a profile for code under test is as easy as enabling one fo the flags below [-cpuprofile, etc...]. Be careful when using more than one flag at a time, however: the machinery for gathering one kind of profile may skew the results of others.

Do you have more info on that? It could be possible that the info is outdated or that this kind of benchmark may be executed just once.

Comment thread pkg/tracer/tracer.go Outdated
// initialize the Tracer
t := &Tracer{
transport: NewHTTPTransport(defaultDeliveryURL),
flushTicker: time.NewTicker(flushInterval),

@LeoCavaille LeoCavaille Sep 15, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you just use the ticker in one method, not sure it's necessary to make it an attribute of the Tracer? Just instantiate and use it in that method?

for range time.Tick()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, totally correct and thanks for catching that. Totally forget to remove that because it was related to the previous approach (multiple channels and go routines that can be stopped). Furthermore our goroutine never stops so there isn't any kind of "leak" for loosing the reference.

@palazzem palazzem force-pushed the palazzem/trace-span-structs branch from c0b1074 to 78989e1 Compare September 15, 2016 12:56
@palazzem palazzem changed the title [WIP] tracer, span and the high-level API [core] Implementation of the tracing package Sep 15, 2016
@palazzem palazzem merged commit 7b3a665 into master Sep 15, 2016
@palazzem palazzem deleted the palazzem/trace-span-structs branch September 15, 2016 14:02
Comment thread pkg/tracer/span.go
@@ -0,0 +1,121 @@
package tracer

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rm the pkg directory

Comment thread pkg/tracer/tracer.go

// Tracer is the common struct we use to collect, buffer
type Tracer struct {
enabled int32 // acts as bool to define if the Tracer is enabled or not

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use a bool here

Comment thread glide.yaml
import:
- package: github.com/stretchr/testify
version: ^1.1.3
- package: github.com/cihub/seelog

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

drop this dependency

Comment thread pkg/tracer/tracer_test.go
// Mock Transport
type DummyTransport struct{}

func (t *DummyTransport) Send(spans []*Span) error { return nil }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the dummy transport should encode the spans to make sure the test data in sane

Comment thread pkg/tracer/tracer_test.go
parent := tracer.NewSpan("pylons.request", "pylons", "/")
child := tracer.NewChildSpan("redis.command", parent)
assert.Equal(child.ParentID, parent.SpanID)
assert.Equal(child.TraceID, parent.TraceID)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please test the right thigs are inherited

Comment thread pkg/tracer/tracer.go
}

// child that is correctly configured
return newSpan(name, parent.Service, parent.Resource, spanID, parent.TraceID, parent.SpanID, parent.tracer)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it shouldn't be inheriting the resource. only the service & ids (see python client)

Comment thread pkg/tracer/context.go

type datadogContextKey struct{}

var datadogActiveSpanKey = datadogContextKey{}

@clutchski clutchski Sep 15, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using an empty struct as the key is kinda crazy. is this idiomatic? why not "datadog_trace_span"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found that in the opentracing-go client. I'm not an expert here but yes.... think a string is enough?

Comment thread pkg/tracer/encoder.go
// Encode returns a byte array related to the marshalling
// of a list of spans.
func (e *JSONEncoder) Encode(spans []*Span) ([]byte, error) {
return json.Marshal(spans)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is really slow let's use the encoder pool from dogtrace. been down this road :)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Comment thread pkg/tracer/transport.go
return err
}

defer response.Body.Close()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to defer at the end of a function. needless perf cost. you're already at the end

@palazzem palazzem restored the palazzem/trace-span-structs branch September 15, 2016 16:18
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jan 27, 2026
This PR updates the behavior of `WithMaxQuerySize` when `max=0` to avoid attaching the query tag entirely. This is more intuitive ("max query size of zero") and gives folks a way to disable serializing the command entirely.
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request May 19, 2026
…SpanOperation (#4766)

### What does this PR do?

Fixes a memory leak in AppSec when running under orchestrion, plus the related listener-cleanup asymmetry that compounds it. Two small, independent changes in two files:

1. **`ServiceEntrySpanOperation.Finish()` now calls `dyngo.FinishOperation`.**
   The op was pushed onto orchestrion's per-goroutine GLS stack by `StartAndRegisterOperation`, but its `Finish()` only flushed JSON tags and never invoked `dyngo.FinishOperation` — and `FinishOperation` is the **only** caller of `orchestrion.GLSPopValue`. On reused goroutines (HTTP keep-alive workers, gRPC handlers, worker pools), the parent op accumulated on the GLS stack one entry per request, retaining the span and the data-listener closures registered by `AppsecSpanTransport.OnServiceEntryStart`.

   The new call is **`defer`-ed**, not appended, so the GLS pop and `disable()` still run if a `TagSetter.SetTag` implementation panics during the tag-flush loop — otherwise the panic path would silently re-introduce the same leak.

2. **`operation.disable()` now also clears `dataBroadcaster.listeners`.**
   `disable()` cleared `eventRegister.listeners` but left its sibling `dataBroadcaster.listeners` populated — an asymmetry between two structurally identical listener maps, both populated by the same `addXListener` family. On ops that are properly finished, this only delays reclamation; combined with the GLS retention above, it amplifies the leak by keeping data-listener closures alive for every retained parent op.

### Motivation

Fixes #4763.

That issue covers fix (1) — the `ServiceEntrySpanOperation` GLS leak — and includes the same `MockGLS()`-based reproduction approach used in this PR's verification.

This PR adds fix (2) on top: the `dataBroadcaster` cleanup asymmetry was found while tracing the same retention path and is responsible for amplifying the heap impact of (1). The two are independent — (1) removes the leak on the orchestrion path; (2) removes the asymmetry that would amplify any future regression of the same shape.

### Root cause walkthrough

**Start path — both ops are registered:**

```
waf.StartContextOperation
 ├── trace.StartServiceEntrySpanOperation
 │     └── dyngo.StartAndRegisterOperation(entrySpanOp)   → GLS push #1 (parent)
 └── dyngo.StartAndRegisterOperation(contextOp)            → GLS push #2 (child)
```

**Finish path (before this PR) — only the child is unwound:**

```go
// internal/appsec/emitter/waf/context.go
func (op *ContextOperation) Finish() {
    dyngo.FinishOperation(op, ContextRes{})   // pops child from GLS
    op.ServiceEntrySpanOperation.Finish()      // flushes tags only — no FinishOperation
}
```

```go
// instrumentation/appsec/trace/service_entry_span.go (before)
func (op *ServiceEntrySpanOperation) Finish() {
    // ranges over op.jsonTags and calls span.SetTag — does NOT call dyngo.FinishOperation
}
```

Per request on the same goroutine, the GLS stack grows by one. `waf.ContextOperation.Finish()` is **unchanged** by this PR — its existing call sequence (pop child, then call parent's `Finish`) now correctly drives both lifecycles.

### Reproduction

Two reproduction tests are not bundled in the PR (kept the diff minimal at 2 files / +16 / -1), but both pass at HEAD and fail when the corresponding fix is reverted.

**Test 1 — GLS stack returns to baseline.**

A note for reviewers running this locally: `orchestrion.Enabled()` is flipped by the orchestrion source-transform tool at build time, not by `-tags orchestrion`. Under plain `go test`, `CtxWithValue` therefore skips the GLS push branch and `GLSStackDepth()` always returns 0 — a test relying on the build tag alone would pass vacuously. The repo's `orchestrion.MockGLS()` (`internal/orchestrion/mock_gls.go`) installs a real `contextStack` and flips `enabled=true` for the duration of the test, which is what #4763's reproduction also uses.

```go
package waf_test

import (
    "context"
    "testing"

    "github.com/DataDog/dd-trace-go/v2/instrumentation/appsec/trace"
    "github.com/DataDog/dd-trace-go/v2/internal/appsec/emitter/waf"
    "github.com/DataDog/dd-trace-go/v2/internal/orchestrion"
)

func TestGLSStackReturnsToBaseline(t *testing.T) {
    defer orchestrion.MockGLS()()

    ctx := context.Background()
    baseline := orchestrion.GLSStackDepth()

    const iterations = 100
    for i := 0; i < iterations; i++ {
        op, _ := waf.StartContextOperation(ctx, trace.NoopTagSetter{})
        op.Finish()
    }

    if got := orchestrion.GLSStackDepth(); got != baseline {
        t.Errorf("GLS leaked: started at %d, now %d after %d cycles", baseline, got, iterations)
    }
}
```

Observed when commit 1 of this PR is reverted: `GLS leaked: started at 0, now 100 after 100 cycles` — exactly +1 entry per request, matching the predicted leak rate. Stable at baseline with the fix applied.

**Test 2 — `disable()` must clear `dataBroadcaster`.**

```go
package dyngo

import "testing"

type repEvt struct{ val string }

func TestDataBroadcasterNotClearedOnDisable(t *testing.T) {
    op := &operation{}
    addDataListener(&op.dataBroadcaster, DataListener[repEvt](func(e repEvt) {}))
    addDataListener(&op.dataBroadcaster, DataListener[repEvt](func(e repEvt) {}))
    addDataListener(&op.dataBroadcaster, DataListener[repEvt](func(e repEvt) {}))

    op.disable()

    op.dataBroadcaster.mu.RLock()
    defer op.dataBroadcaster.mu.RUnlock()
    total := 0
    for _, ls := range op.dataBroadcaster.listeners {
        total += len(ls)
    }
    if total != 0 {
        t.Errorf("dataBroadcaster has %d listeners after disable(), want 0", total)
    }
}
```

Observed when commit 2 is reverted: `dataBroadcaster has 3 listeners after disable(), want 0`. Passes at HEAD.

Happy to include either or both tests in the PR if reviewers want — they were left out only to keep the diff narrowly scoped to the fix.

### Memory evidence

On a long-lived process with AppSec + orchestrion enabled and steady HTTP traffic:

- AppSec on, before fix: monotonic heap growth that scales with request count.
- AppSec off: flat heap.
- AppSec on, with both fixes: flat heap.

Heap-live-size profile retention path matches the GLS analysis:

```
dyngo.addDataListener
  ← dyngo.OnData
    ← AppsecSpanTransport.OnServiceEntryStart
      ← dyngo.StartAndRegisterOperation               (called once per HTTP request)
```

The path terminates in `StartAndRegisterOperation` with no matching `FinishOperation` — consistent with the GLS-pin hypothesis.

### Files touched

- `instrumentation/appsec/trace/service_entry_span.go` — +9 / -1
- `instrumentation/appsec/dyngo/operation.go` — +7 / -0

Total: 2 files, +16 / -1.

### Reviewer's Checklist

- [ ] Changed code has unit tests for its functionality at or near 100% coverage.
- [ ] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag.
- [ ] There is a benchmark for any new code, or changes to existing code.
- [ ] If this interacts with the agent in a new way, a system test has been added.
- [x] New code is free of linting errors. You can check this by running `make lint` locally.
- [x] New code doesn't break existing tests. You can check this by running `make test` locally.
- [ ] Add an appropriate team label so this PR gets put in the right place for the release notes.
- [x] All generated files are up to date. You can check this by running `make generate` locally.
- [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally.


Co-authored-by: e-n-0 <flavien.darche@datadoghq.com>
Co-authored-by: dario.castane <dario.castane@datadoghq.com>
hannahkm pushed a commit that referenced this pull request May 22, 2026
…SpanOperation (#4766)

### What does this PR do?

Fixes a memory leak in AppSec when running under orchestrion, plus the related listener-cleanup asymmetry that compounds it. Two small, independent changes in two files:

1. **`ServiceEntrySpanOperation.Finish()` now calls `dyngo.FinishOperation`.**
   The op was pushed onto orchestrion's per-goroutine GLS stack by `StartAndRegisterOperation`, but its `Finish()` only flushed JSON tags and never invoked `dyngo.FinishOperation` — and `FinishOperation` is the **only** caller of `orchestrion.GLSPopValue`. On reused goroutines (HTTP keep-alive workers, gRPC handlers, worker pools), the parent op accumulated on the GLS stack one entry per request, retaining the span and the data-listener closures registered by `AppsecSpanTransport.OnServiceEntryStart`.

   The new call is **`defer`-ed**, not appended, so the GLS pop and `disable()` still run if a `TagSetter.SetTag` implementation panics during the tag-flush loop — otherwise the panic path would silently re-introduce the same leak.

2. **`operation.disable()` now also clears `dataBroadcaster.listeners`.**
   `disable()` cleared `eventRegister.listeners` but left its sibling `dataBroadcaster.listeners` populated — an asymmetry between two structurally identical listener maps, both populated by the same `addXListener` family. On ops that are properly finished, this only delays reclamation; combined with the GLS retention above, it amplifies the leak by keeping data-listener closures alive for every retained parent op.

### Motivation

Fixes #4763.

That issue covers fix (1) — the `ServiceEntrySpanOperation` GLS leak — and includes the same `MockGLS()`-based reproduction approach used in this PR's verification.

This PR adds fix (2) on top: the `dataBroadcaster` cleanup asymmetry was found while tracing the same retention path and is responsible for amplifying the heap impact of (1). The two are independent — (1) removes the leak on the orchestrion path; (2) removes the asymmetry that would amplify any future regression of the same shape.

### Root cause walkthrough

**Start path — both ops are registered:**

```
waf.StartContextOperation
 ├── trace.StartServiceEntrySpanOperation
 │     └── dyngo.StartAndRegisterOperation(entrySpanOp)   → GLS push #1 (parent)
 └── dyngo.StartAndRegisterOperation(contextOp)            → GLS push #2 (child)
```

**Finish path (before this PR) — only the child is unwound:**

```go
// internal/appsec/emitter/waf/context.go
func (op *ContextOperation) Finish() {
    dyngo.FinishOperation(op, ContextRes{})   // pops child from GLS
    op.ServiceEntrySpanOperation.Finish()      // flushes tags only — no FinishOperation
}
```

```go
// instrumentation/appsec/trace/service_entry_span.go (before)
func (op *ServiceEntrySpanOperation) Finish() {
    // ranges over op.jsonTags and calls span.SetTag — does NOT call dyngo.FinishOperation
}
```

Per request on the same goroutine, the GLS stack grows by one. `waf.ContextOperation.Finish()` is **unchanged** by this PR — its existing call sequence (pop child, then call parent's `Finish`) now correctly drives both lifecycles.

### Reproduction

Two reproduction tests are not bundled in the PR (kept the diff minimal at 2 files / +16 / -1), but both pass at HEAD and fail when the corresponding fix is reverted.

**Test 1 — GLS stack returns to baseline.**

A note for reviewers running this locally: `orchestrion.Enabled()` is flipped by the orchestrion source-transform tool at build time, not by `-tags orchestrion`. Under plain `go test`, `CtxWithValue` therefore skips the GLS push branch and `GLSStackDepth()` always returns 0 — a test relying on the build tag alone would pass vacuously. The repo's `orchestrion.MockGLS()` (`internal/orchestrion/mock_gls.go`) installs a real `contextStack` and flips `enabled=true` for the duration of the test, which is what #4763's reproduction also uses.

```go
package waf_test

import (
    "context"
    "testing"

    "github.com/DataDog/dd-trace-go/v2/instrumentation/appsec/trace"
    "github.com/DataDog/dd-trace-go/v2/internal/appsec/emitter/waf"
    "github.com/DataDog/dd-trace-go/v2/internal/orchestrion"
)

func TestGLSStackReturnsToBaseline(t *testing.T) {
    defer orchestrion.MockGLS()()

    ctx := context.Background()
    baseline := orchestrion.GLSStackDepth()

    const iterations = 100
    for i := 0; i < iterations; i++ {
        op, _ := waf.StartContextOperation(ctx, trace.NoopTagSetter{})
        op.Finish()
    }

    if got := orchestrion.GLSStackDepth(); got != baseline {
        t.Errorf("GLS leaked: started at %d, now %d after %d cycles", baseline, got, iterations)
    }
}
```

Observed when commit 1 of this PR is reverted: `GLS leaked: started at 0, now 100 after 100 cycles` — exactly +1 entry per request, matching the predicted leak rate. Stable at baseline with the fix applied.

**Test 2 — `disable()` must clear `dataBroadcaster`.**

```go
package dyngo

import "testing"

type repEvt struct{ val string }

func TestDataBroadcasterNotClearedOnDisable(t *testing.T) {
    op := &operation{}
    addDataListener(&op.dataBroadcaster, DataListener[repEvt](func(e repEvt) {}))
    addDataListener(&op.dataBroadcaster, DataListener[repEvt](func(e repEvt) {}))
    addDataListener(&op.dataBroadcaster, DataListener[repEvt](func(e repEvt) {}))

    op.disable()

    op.dataBroadcaster.mu.RLock()
    defer op.dataBroadcaster.mu.RUnlock()
    total := 0
    for _, ls := range op.dataBroadcaster.listeners {
        total += len(ls)
    }
    if total != 0 {
        t.Errorf("dataBroadcaster has %d listeners after disable(), want 0", total)
    }
}
```

Observed when commit 2 is reverted: `dataBroadcaster has 3 listeners after disable(), want 0`. Passes at HEAD.

Happy to include either or both tests in the PR if reviewers want — they were left out only to keep the diff narrowly scoped to the fix.

### Memory evidence

On a long-lived process with AppSec + orchestrion enabled and steady HTTP traffic:

- AppSec on, before fix: monotonic heap growth that scales with request count.
- AppSec off: flat heap.
- AppSec on, with both fixes: flat heap.

Heap-live-size profile retention path matches the GLS analysis:

```
dyngo.addDataListener
  ← dyngo.OnData
    ← AppsecSpanTransport.OnServiceEntryStart
      ← dyngo.StartAndRegisterOperation               (called once per HTTP request)
```

The path terminates in `StartAndRegisterOperation` with no matching `FinishOperation` — consistent with the GLS-pin hypothesis.

### Files touched

- `instrumentation/appsec/trace/service_entry_span.go` — +9 / -1
- `instrumentation/appsec/dyngo/operation.go` — +7 / -0

Total: 2 files, +16 / -1.

### Reviewer's Checklist

- [ ] Changed code has unit tests for its functionality at or near 100% coverage.
- [ ] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag.
- [ ] There is a benchmark for any new code, or changes to existing code.
- [ ] If this interacts with the agent in a new way, a system test has been added.
- [x] New code is free of linting errors. You can check this by running `make lint` locally.
- [x] New code doesn't break existing tests. You can check this by running `make test` locally.
- [ ] Add an appropriate team label so this PR gets put in the right place for the release notes.
- [x] All generated files are up to date. You can check this by running `make generate` locally.
- [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally.


Co-authored-by: e-n-0 <flavien.darche@datadoghq.com>
Co-authored-by: dario.castane <dario.castane@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants