Skip to content

Remote debug metricbeat running on docker/k8s from your laptop#31748

Merged
gsantoro merged 19 commits intoelastic:mainfrom
gsantoro:feature/remote-debugger
Jun 15, 2022
Merged

Remote debug metricbeat running on docker/k8s from your laptop#31748
gsantoro merged 19 commits intoelastic:mainfrom
gsantoro:feature/remote-debugger

Conversation

@gsantoro
Copy link
Copy Markdown
Contributor

@gsantoro gsantoro commented May 25, 2022

What does this PR do?

Allow to remote debug metricbeat running on kubernetes from VisualStudioCode

Why is it important?

Remote debugging with breakpoints is far superior to fmt.printLn

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files

Author's Checklist

  • try to remote debug running on kind on your laptop

How to test this PR locally

create a local file at .vscode/launch.json to run the debugger from VisualStudioCode on your laptop

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Connect to server",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "debugAdapter": "dlv-dap",
            "port": 56268,
            "host": "127.0.0.1",
            "showLog": true,
            "trace": "trace",
            "cwd": "${workspaceFolder}",
            "substitutePath": [
                { 
	                "from": "${workspaceFolder}",
		   # NOTE: REPLACE `<absolute path>` with the absolute path of the root folder
                    "to": "<absolute path>"   
                }
            ]
        }
      
    ]
}

Steps to run on docker:

  1. Move into metricbeat folder. Note: For some reason even if we are building in a subfolder, the substitutePath.to is still pointing to our root folder.
cd metricbeat
  1. cross build image for linux
GOOS=linux GOARCH=amd64 go build -gcflags "-N -l" -o metricbeat main.go
  1. buld docker container
docker build -t metricbeat-debugger-image -f Dockerfile.debug .
  1. run container
docker run -p 56268:56268 --network elastic-package-stack_default metricbeat-debugger-image
  1. Run debugger from VisualStudio Code via .vscode/launch.json. Remember to add first some breakpoints

Steps to run on kubernetes:

Steps from 0 to 2 (included) are the same as Steps to run on docker
0. Move into metricbeat folder. Note: For some reason even if we are building in a subfolder, the substitutePath.to is still pointing to our root folder.

cd metricbeat
  1. cross build image for linux
GOOS=linux GOARCH=amd64 go build -gcflags "-N -l" -o metricbeat main.go
  1. buld docker container
docker build -t metricbeat-debugger-image -f Dockerfile.debug .
  1. load image into Kind in order to run on kubernetes
kind load docker-image --name kind-v1.23.5 metricbeat-debugger-image:latest
  1. Edit deploy/kubernetes/metricbeat-kubernetes.yaml with these changes
containers:
- name: metricbeat
  # image: docker.elastic.co/beats/metricbeat:8.2.0
  image: metricbeat-debugger-image:latest
  imagePullPolicy: Never
  args: [
    "-c", "/etc/metricbeat.yml",
    "-e",
    "-system.hostfs=/hostfs",
  ]
  ports:
    - containerPort: 56268
      hostPort: 56268
      protocol: TCP

Namely you need:

  • change the docker image used
  • add imagePullPolicy to pull the image from inside Kind
  • add a ports to expose the port in order to remote debug from laptop
  1. Port forward from k8s to localhost
kubectl port-forward <pod-name> 56268:56268

where <pod-name> is the name of the pod running on k8s

  1. Run debugger from VisualStudio Code via .vscode/launch.json. Remember to add first some breakpoints. For example you can put a breakpoint at metricbeat/cmd/root.go at line 74 to stop at the very beginning of the metricbeat command

Use cases

  • Run remote debugger on docker
  • run remote debugger on kubernetes (on Kind)

@gsantoro gsantoro added enhancement Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team labels May 25, 2022
@gsantoro gsantoro requested a review from a team as a code owner May 25, 2022 10:29
@gsantoro gsantoro requested review from leehinman and rdner and removed request for a team May 25, 2022 10:29
@gsantoro gsantoro self-assigned this May 25, 2022
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 25, 2022
@gsantoro gsantoro requested a review from MichaelKatsoulis May 25, 2022 10:33
@gsantoro gsantoro requested a review from a team as a code owner May 25, 2022 10:38
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented May 25, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-06-15T14:48:32.887+0000

  • Duration: 56 min 41 sec

Test stats 🧪

Test Results
Failed 0
Passed 3527
Skipped 873
Total 4400

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@gsantoro
Copy link
Copy Markdown
Contributor Author

gsantoro commented May 25, 2022

/test

@gsantoro gsantoro requested a review from gizas May 30, 2022 09:38
rdner
rdner requested changes May 31, 2022
View reviewed changes
@gsantoro gsantoro mentioned this pull request May 31, 2022
Closed
@gsantoro gsantoro mentioned this pull request May 31, 2022
Merged
8 tasks
@gsantoro gsantoro requested a review from rdner June 7, 2022 14:11
tetianakravchenko
tetianakravchenko approved these changes Jun 13, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@tetianakravchenko tetianakravchenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gsantoro thank you for the detailed readme! I was able to run remote debugger locally

just one comment:
from my understanding this is mainly to debug metricbeat on k8s or in docker, should maybe then be added a comment in dockerfile/ or adjusted dockerfile name to point it out?

@gsantoro @tetianakravchenko
Update metricbeat/module/kubernetes/_meta/remote-debugger/README.md
713fde5 
Co-authored-by: Tetiana Kravchenko <tanya.kravchenko.v@gmail.com>
@gsantoro
Copy link
Copy Markdown
Contributor Author

gsantoro commented Jun 13, 2022

hey @tetianakravchenko, thanks for reviewing my PR, can you please clarify what you mean by?

just one comment:
from my understanding this is mainly to debug metricbeat on k8s or in docker, should maybe then be added a comment in dockerfile/ or adjusted dockerfile name to point it out?

Let me see if I can make this clearer.

  1. The dockerfile is under metricbeat folder and the readme mentions that this is only for metricbeat (at least for now)
  2. Given that we are going to move away from beats and use elastic-agent going forward, I'm planning to find a solution for elastic-agent as well. I am also considering if I could use Skaffold as an alternative to also provide hot reloading in the future. So this is just a work in progress but still working.

@gsantoro
Copy link
Copy Markdown
Contributor Author

gsantoro commented Jun 13, 2022

/test

@tetianakravchenko
Copy link
Copy Markdown
Contributor

tetianakravchenko commented Jun 15, 2022

@gsantoro

Let me see if I can make this clearer.

The dockerfile is under metricbeat folder and the readme mentions that this is only for metricbeat (at least for now)
Given that we are going to move away from beats and use elastic-agent going forward, I'm planning to find a solution for elastic-agent as well. I am also considering if I could use Skaffold as an alternative to also provide hot reloading in the future. So this is just a work in progress but still working.

my point was, that now we have a readme located in k8s module folder (metricbeat/module/kubernetes/_meta/remote-debugger/README.md) that explains what is the use of dockerfile, that is located on top level metricbeat/Dockerfile.debug. Without context (like a comment in dockerfile) it might be confusing to understand the usage of this dockerfile.
it is not a blocker, feel free to merge this pr

@gsantoro gsantoro merged commit 0c60aa8 into elastic:main Jun 15, 2022
@gsantoro gsantoro deleted the feature/remote-debugger branch June 15, 2022 15:53
@gsantoro gsantoro mentioned this pull request Jun 21, 2022
Closed
13 tasks
@gsantoro gsantoro mentioned this pull request Jul 12, 2022
Merged
13 tasks
@gsantoro gsantoro mentioned this pull request Nov 17, 2022
Closed
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
@gsantoro @chrisberkhout
Remote debug metricbeat running on docker/k8s from your laptop (#31748)
5459908 
* extra Dockerfile.debug with instructions on how to use remote debugging to debug metricbeat running on k8s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rdner rdner rdner approved these changes

@MichaelKatsoulis MichaelKatsoulis MichaelKatsoulis approved these changes

@tetianakravchenko tetianakravchenko tetianakravchenko approved these changes

@leehinman leehinman Awaiting requested review from leehinman leehinman is a code owner automatically assigned from elastic/elastic-agent-data-plane

@gizas gizas Awaiting requested review from gizas

Assignees

@gsantoro gsantoro

Labels

enhancement Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@gsantoro @elasticmachine @tetianakravchenko @rdner @MichaelKatsoulis