parca debug-info upload by korniltsev-grafanista · Pull Request #4648 · grafana/pyroscope

korniltsev-grafanista · 2025-11-28T07:53:53Z

This PR copies the code from https://github.com/parca-dev/parca/tree/main/pkg/debuginfo into pyroscope and integrates with the existing distributors and symbolizer

The parca's debuginfo uploads debug info binaries to a separate bucket subdir.
The symbolizer later uses the new subdir as the source for debug info binaries as an alternative to debuginfod.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add debug-level logging to help troubleshoot debuginfo upload issues: - Log ShouldInitiateUpload results (build_id, decision, reason) - Log Upload gRPC results (success with size, or failure with error) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Debuginfod fetching is now handled in the symbolizer instead of in the parca debuginfo store. This removes the debuginfod client implementation, related tests, and test fixtures from the parca/debuginfo package. Also removes tenant ID handling from the debuginfo store since it's not needed for this use case. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…test Refactor the store test to use an HTTP/2 server with h2c and the util.AuthenticateUser middleware instead of a direct gRPC server. This better reflects the production setup where requests go through HTTP middleware for tenant authentication. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

pkg/parca/debuginfo/metadata.go

pkg/parca/debuginfo/store.go

pkg/symbolizer/symbolizer.go

marcsanmi · 2026-01-20T17:59:41Z

pkg/debuginfo/parcaglue.go

+
+	t := noop.Tracer{}
+	l = log.With(l, "component", "debug-info")
+	bucket = objstore.NewPrefixedBucket(bucket, symbolizer.BucketPrefixParcaDebugInfo)


I think we don't have a cleanup mechanism to control unbounded growth (neither for symbolizer path). Could it be an issue?

I think it's fine to keep lidia's indefinitely. And preferably to remove the debug information files once they are converted to lidia. All of this is hard to implement in a stateless manner in distributors, so I propose to postpone these decision until we have a proper place/way to implement this https://github.com/grafana/pyroscope-squad/issues/687

pkg/parca/cache/cache.go

Co-authored-by: Marc Sanmiquel <marcsanmiquel@gmail.com>

Test creates a lidia table from /proc/self/exe using only gopclntab (symtab disabled), then resolves the test function's own address and verifies the name matches. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

#### PR Description This PR introduces debug information upload using parca's API implemented in grafana/pyroscope#4648 on the pyroscope This feature is implemented in two places: the profiler and the proxy ```go type debuginfo.Appender interface { Upload(j UploadJob) // for the profiler: fanout to all the nested children Client() debuginfogrpc.DebuginfoServiceClient // for the proxy - retrieve the first grpc client available and proxy data to the client } ``` The profile uploader actually lives in the `pyroscope.write` component to support per-write component uploads(ugh who needs this? no-one!) , but this is an implementation detail and it's hidden from the users. The proxy only can proxy data to the one client - it can not fan-out. The `parca` package is copied from the parca's repo with some adjustments (`logrus` -> `logkit`, the grpc client is passed as an argument to the upload routine instead of creating and owning the grpc client inside the uploader) #### Which issue(s) this PR fixes   #### Notes to the Reviewer #### PR Checklist  - [ ] CHANGELOG.md updated - [x] Documentation added - [x] Tests updated - [ ] Config converters updated --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

simonswine

I have not tired this end-to-end but I think this looks mostly ready. I have a concern about the exposing of the service. Wonder how you feel about it.

simonswine · 2026-01-28T10:22:31Z

pkg/api/api.go

+func (a *API) RegisterDebugInfo(d *grpc.Server, limits *validation.Overrides) {
+	const (
+		DebuginfoService_Upload_FullMethodName               = "/parca.debuginfo.v1alpha1.DebuginfoService/Upload"
+		DebuginfoService_ShouldInitiateUpload_FullMethodName = "/parca.debuginfo.v1alpha1.DebuginfoService/ShouldInitiateUpload"
+		DebuginfoService_InitiateUpload_FullMethodName       = "/parca.debuginfo.v1alpha1.DebuginfoService/InitiateUpload"
+		DebuginfoService_MarkUploadFinished_FullMethodName   = "/parca.debuginfo.v1alpha1.DebuginfoService/MarkUploadFinished"
+	)
+	debugInfoGRPCOptions := []RegisterOption{
+		a.WithAuthMiddleware(),
+	}
+	a.RegisterRoute(DebuginfoService_Upload_FullMethodName, d, debugInfoGRPCOptions...)
+	a.RegisterRoute(DebuginfoService_ShouldInitiateUpload_FullMethodName, d, debugInfoGRPCOptions...)
+	a.RegisterRoute(DebuginfoService_InitiateUpload_FullMethodName, d, debugInfoGRPCOptions...)
+	a.RegisterRoute(DebuginfoService_MarkUploadFinished_FullMethodName, d, debugInfoGRPCOptions...)
+}


I am a bit worried of our mix of GRPC and Connect-Go. None of this really your fault in this PR, but we have currently:

Cell internal comms which are a mix of GRPC (From Pyroscope v2 project) and connect-go (before v2)

Public endpoints are all connect-go (accepting GRPC) or Pyroscope legacy and we do have a endpoint GRPC endpoint for compatibility with OTLP.

So after your PR we would require a client that send both connect-go for Push and GRPC for the uploads. This feels making the above situation worse.

How do you/others feel about this? Is it worth copying the parca proto into api/debuginfo and generate our own handlers clients?

If we would do that we can reuse the connect handlers and the client would be complete within the api/ package. The endpoints would also work with curl json/grpc/connect and we would be able to reuse e.g. the API docs generation.

At first I was against it because I wanted to preserve as much parca code as possible in genuine state, so that it is diffed and updated in case needed easily. I later had to modify quite a lot of code, so now I am actually in favor of
Yes, let's do this. The only problem is that I've already merged client code =/
I will try to either update the client and backport tomorrow, or disable the new functionality and aim for the next release.

simonswine

Thanks for working on this.

I think this is good enough for an initial MVP

simonswine · 2026-02-13T11:07:26Z

pkg/symbolizer/symbolizer_test.go

 	Limits   Limits
 }

-func newSymbolizerTest(t *testing.T, inp *symbolizerInputs) (*Symbolizer, *mocksymbolizer.MockDebuginfodClient, *mockobjstore.MockBucket) {


This needs to be rebased before merge. Maybe you just forget to push?

marcsanmi · 2026-02-13T16:42:24Z

pkg/symbolizer/symbolizer.go

+	return s.fetchFromDebuginfod(ctx, buildID)
+}
+
+func (s *Symbolizer) fetchFromParca(ctx context.Context, buildID string) (io.ReadCloser, error) {


maybe (or something similar)?

Suggested change

func (s *Symbolizer) fetchFromParca(ctx context.Context, buildID string) (io.ReadCloser, error) {

func (s *Symbolizer) fetchFromUploadedDebugInfo(ctx context.Context, buildID string) (io.ReadCloser, error) {

marcsanmi · 2026-02-13T16:45:34Z

pkg/debuginfo/store.go

+	if err := s.bucket.Upload(ctx, ObjectPath(tenantID, id), r); err != nil {
+		return status.Error(codes.Internal, fmt.Errorf("upload debuginfo: %w", err).Error())
+	}
+	md.FinishedAt = timestamppb.New(time.Now())


Should we update the state to ObjectMetadata_STATE_UPLOADED around here?

marcsanmi · 2026-02-13T16:47:52Z

pkg/debuginfo/store.go

+		return nil, fmt.Errorf("first message expected to be init")
+	}
+	if init.File == nil {
+		return nil, fmt.Errorf("init.FileData == nil")


Suggested change

return nil, fmt.Errorf("init.FileData == nil")

return nil, fmt.Errorf("init.File == nil")

marcsanmi · 2026-02-13T16:49:27Z

pkg/debuginfo/store.go

+
+	r := debuginforeader.New(ctx, readChunksFromStream(stream))
+	if err := s.bucket.Upload(ctx, ObjectPath(tenantID, id), r); err != nil {
+		return status.Error(codes.Internal, fmt.Errorf("upload debuginfo: %w", err).Error())


This seems mixed grpc and connect errors.

marcsanmi · 2026-02-13T17:04:44Z

api/debuginfo/v1/debuginfo.proto

+message FileMetadata {
+  enum DebuginfoType {
+    DEBUGINFO_TYPE_EXECUTABLE_FULL = 0;
+    DEBUGINFO_TYPE_EXECUTABLE_NO_TEXT = 1;
+    //  DEBUGINFO_TYPE_LIDIA = 1;
+    //  DEBUGINFO_TYPE_BLAZESYM = 2;
+  }
+
+  string GNU = 1;
+  string Golang = 2;
+  // optional libpf.FileID rom the otel profiler
+  string OpenTelemetry = 3;
+  string Name = 4;
+  DebuginfoType type = 5;
+}


Suggested change

message FileMetadata {

enum DebuginfoType {

DEBUGINFO_TYPE_EXECUTABLE_FULL = 0;

DEBUGINFO_TYPE_EXECUTABLE_NO_TEXT = 1;

// DEBUGINFO_TYPE_LIDIA = 1;

// DEBUGINFO_TYPE_BLAZESYM = 2;

}

string GNU = 1;

string Golang = 2;

// optional libpf.FileID rom the otel profiler

string OpenTelemetry = 3;

string Name = 4;

DebuginfoType type = 5;

}

message FileMetadata {

enum Type {

TYPE_UNSPECIFIED = 0; // several times used as a good practise on enums, feel free to omit it

TYPE_EXECUTABLE_FULL = 1;

TYPE_EXECUTABLE_NO_TEXT = 2;

}

// GNU build ID.

string gnu_build_id = 1;

// Go build ID.

string go_build_id = 2;

// OpenTelemetry profiler file ID (libpf.FileID).

string otel_file_id = 3;

// Original binary file name.

string name = 4;

// Type of the debug info file.

Type type = 5;

}

korniltsev-grafanista · 2026-02-16T07:27:41Z

Just wanted to clarify, the PR was not ready for review, only the new protobufs and overall intention to use the new protobufs.

I also think we should have an idea how to move forward with infinite data growth (both lidia and debug info)

korniltsev-grafanista force-pushed the debuginfo-upload branch 5 times, most recently from d202551 to 8577f33 Compare January 16, 2026 07:36

parca symbol upload

5c053a7

korniltsev-grafanista force-pushed the debuginfo-upload branch from 8577f33 to 5c053a7 Compare January 16, 2026 07:37

test: add symbolizer test for parca bucket debuginfo source

3748d11

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

korniltsev-grafanista changed the title ~~WIP parca debug-info upload~~ parca debug-info upload Jan 16, 2026

korniltsev-grafanista marked this pull request as ready for review January 16, 2026 08:05

korniltsev-grafanista requested review from a team, aleks-p, alsoba13, marcsanmi and simonswine as code owners January 16, 2026 08:05

korniltsev-grafanista and others added 5 commits January 19, 2026 05:45

make go/mod

ebac661

make fmt

de30ec7

marcsanmi reviewed Jan 20, 2026

View reviewed changes

korniltsev-grafanista and others added 5 commits January 21, 2026 15:29

Update pkg/parca/debuginfo/metadata.go

6996be3

Co-authored-by: Marc Sanmiquel <marcsanmiquel@gmail.com>

remove writepath options from the new debuginfo handlers

a6e1367

level.Error

51b6c96

cleanup

c4a913d

remove cache

73a9406

korniltsev-grafanista mentioned this pull request Jan 22, 2026

feat: Add parcas symbols upload to pyroscope.ebpf grafana/alloy#4948

Merged

4 tasks

korniltsev-grafanista added 3 commits January 22, 2026 14:51

Merge branch 'main' into debuginfo-upload

ebf62a6

init

e0bb853

add gopclntab

2574f90

korniltsev-grafanista and others added 5 commits January 23, 2026 15:28

add test for gopclntab-only symbol resolution

3d4f860

Test creates a lidia table from /proc/self/exe using only gopclntab (symtab disabled), then resolves the test function's own address and verifies the name matches. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

lint

6abe078

disable unused linter for pclntab.go

03ce701

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/kk/lidiago' into debuginfo-upload

fb78ae9

new file name for lidia files with gopclntab support

298c53f

korniltsev-grafanista force-pushed the debuginfo-upload branch from 005fedc to 298c53f Compare January 26, 2026 06:28

korniltsev-grafanista added 2 commits January 26, 2026 14:57

improve error logging

fe836c7

increase upload size

2abd0d7

korniltsev-grafanista mentioned this pull request Jan 27, 2026

symbolizer: do not keep symbols and debuginfo indefinetely #4792

Open

simonswine reviewed Jan 28, 2026

View reviewed changes

korniltsev-grafanista mentioned this pull request Jan 29, 2026

fix: Remove Parca debug info upload from user configuration grafana/alloy#5391

Merged

3 tasks

grafana-alloybot bot mentioned this pull request Jan 29, 2026

fix: Remove Parca debug info upload from user configuration [backport] grafana/alloy#5395

Merged

3 tasks

korniltsev-grafanista added 2 commits February 9, 2026 16:54

dont use parca

caae3c7

dont use parca

494a4ed

korniltsev-grafanista requested a review from a team as a code owner February 9, 2026 09:54

korniltsev-grafanista added 9 commits February 9, 2026 17:18

generate

8a337e3

generate

69b51c6

Merge remote-tracking branch 'origin/main' into debuginfo-upload

151252a

revert changes

42bce6d

revert changes

187034c

revert changes

ccbc29e

revert changes

3f24dfb

todo

7fcddb8

revert change

21407c6

simonswine approved these changes Feb 13, 2026

View reviewed changes

simonswine reviewed Feb 13, 2026

View reviewed changes

marcsanmi reviewed Feb 13, 2026

View reviewed changes

korniltsev-grafanista marked this pull request as draft February 16, 2026 07:25

	func (s *Symbolizer) fetchFromParca(ctx context.Context, buildID string) (io.ReadCloser, error) {
	func (s *Symbolizer) fetchFromUploadedDebugInfo(ctx context.Context, buildID string) (io.ReadCloser, error) {

	return nil, fmt.Errorf("init.FileData == nil")
	return nil, fmt.Errorf("init.File == nil")

Conversation

korniltsev-grafanista commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

simonswine left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simonswine left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

korniltsev-grafanista commented Feb 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

korniltsev-grafanista commented Nov 28, 2025 •

edited

Loading