You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Please include a minimal but complete recreation of the problem,
including server configuration, agent(s) used, etc. The easier you make it
for us to reproduce it, the more likely that somebody will take the time to
look at it.
Install glibc 2.34
go build the project (already built binary might as well have the bug)
run ./apm-server -c <configuration file>, the configuration file must point to a working Elasticsearch node / cluster
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.564+0200","log.origin":{"file.name":"instance/beat.go","file.line":654},"message":"Home path: [/home/arnaud/Dev/clevercloud/apm-server] Config path: [/home/arnaud/Dev/clevercloud/apm-server] Data path: [/home/arnaud/Dev/clevercloud/apm-server/data] Logs path: [/home/arnaud/Dev/clevercloud/apm-server/logs]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.564+0200","log.origin":{"file.name":"instance/beat.go","file.line":662},"message":"Beat ID: da8fee46-f825-4071-86cd-43796243f7ac","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.565+0200","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.565+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":996},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/home/arnaud/Dev/clevercloud/apm-server","data":"/home/arnaud/Dev/clevercloud/apm-server/data","home":"/home/arnaud/Dev/clevercloud/apm-server","logs":"/home/arnaud/Dev/clevercloud/apm-server/logs"},"type":"apm-server","uuid":"da8fee46-f825-4071-86cd-43796243f7ac"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.565+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1005},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"unknown","libbeat":"8.0.0","time":"0001-01-01T00:00:00.000Z","version":"8.0.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.565+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1008},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":8,"version":"go1.17.1"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.566+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1012},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","boot_time":"2021-09-15T09:02:01+02:00","containerized":false,"name":"exherswag","ip":["127.0.0.1/8","::1/128","192.168.1.6/24","2a01:e34:ec52:4920:a952:2b5f:f139:634c/64","fe80::345e:3214:b582:590e/64"],"kernel_version":"5.13.16+","mac":["2e:88:67:4e:96:5b","40:a3:cc:c2:b1:db"],"os":{"type":"linux","family":"","platform":"exherbo","name":"Exherbo","version":"","major":0,"minor":0,"patch":0},"timezone":"CEST","timezone_offset_sec":7200,"id":"8c53e4b5e43a473dabab6e031c03f534"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.566+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1041},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null},"cwd":"/home/arnaud/Dev/golang/src/github.com/elastic/apm-server","exe":"/home/arnaud/Dev/golang/src/github.com/elastic/apm-server/apm-server","name":"apm-server","pid":965791,"ppid":3091366,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2021-09-24T00:52:52.830+0200"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.566+0200","log.origin":{"file.name":"instance/beat.go","file.line":309},"message":"Setup Beat: apm-server; Version: 8.0.0","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.567+0200","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":102},"message":"elasticsearch url: https://btzojipgx65c22yejkwz-elasticsearch.services.clever-cloud.com:443","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.567+0200","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: exherswag","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.567+0200","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":634},"message":"Registering pipeline callback","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.567+0200","log.origin":{"file.name":"instance/beat.go","file.line":473},"message":"apm-server start running.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":155},"message":"Listening on: [::]:8080","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":125},"message":"Path / added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":125},"message":"Path /assets/v1/sourcemaps added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":125},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":125},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":125},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":125},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":125},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":125},"message":"Path /intake/v2/profile added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":259},"message":"Starting apm-server [unknown built 0001-01-01 00:00:00 +0000 UTC]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":99},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"onboarding","log.origin":{"file.name":"beater/onboarding.go","file.line":34},"message":"Publishing onboarding document","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":115},"message":"Secret token is set, but SSL is not enabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:53.568+0200","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":117},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:54.569+0200","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":143},"message":"Connecting to backoff(elasticsearch(https://btzojipgx65c22yejkwz-elasticsearch.services.clever-cloud.com:443))","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:54.569+0200","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":219},"message":"retryer: send unwait signal to consumer","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-24T00:52:54.569+0200","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":223},"message":" done","service.name":"apm-server","ecs.version":"1.6.0"}
GDB backtrace
Thread 7 "apm-server" received signal SIGABRT, Aborted.
[Switching to LWP 974562]
0x00007ffff7e31ad3 in pthread_kill@GLIBC_2.2.5 () from /usr/x86_64-pc-linux-gnu/lib/libc.so.6
(gdb) bt
#0 0x00007ffff7e31ad3 in pthread_kill@GLIBC_2.2.5 () from /usr/x86_64-pc-linux-gnu/lib/libc.so.6
#1 0x00007ffff7de52f6 in raise () from /usr/x86_64-pc-linux-gnu/lib/libc.so.6
#2 0x00007ffff7dcf7b5 in abort () from /usr/x86_64-pc-linux-gnu/lib/libc.so.6
#3 0x0000000003300069 in fatalf (format=format@entry=0x3f2854c "pthread_create failed: %s") at gcc_fatalf.c:22
#4 0x0000000003300411 in _cgo_sys_thread_start (ts=<optimized out>) at gcc_linux_amd64.c:78
#5 0x000000000147502d in runtime.asmcgocall () at /usr/lib/go/src/runtime/asm_amd64.s:795
#6 0x00007fffc9ffac28 in ?? ()
#7 0x0000000001418727 in runtime.newobject (typ=<optimized out>, ~r1=<optimized out>) at /usr/lib/go/src/runtime/malloc.go:1228
#8 0x0000000000000000 in ?? ()
** Additional information **
This issue started to happen as soon as we switched to glibc 2.34. It did not happen on glibc 2.33. Downgrading glibc from 2.34 to 2.33 resolves the issue, with the exact same binaries. This problem might as well be a golang bug but I'm not really sure and since I don't see any issue on their bug tracker, I'm wondering if we are the only ones hitting this.
I've seen a few projects that also had issue with this new syscall (Docker, Firefox and Chromium). They all mention issues with their sandbox which is based on seccomp. I see the APM project also depend on https://github.com/elastic/go-seccomp-bpf. I couldn't see it really used or even any syscall filtering on the go-seccomp-bpf project as there are on the previously mentionned projets, but maybe I'm overlooking something?
It's also worth to mention that running as root doesn't fix the Operation not permitted error.
Let me know if you need any other information, I'm not a go expert at all. Thanks for any help you could provide, I'll keep looking on my side.
APM Server version (
apm-server version):apm-server version 8.0.0 (amd64), libbeat 8.0.0 [unknown built unknown] (current master but the issue also applies to previous apm-server version)
Description of the problem including expected versus actual behavior:
Since we upgraded to glibc 2.34, the apm-server crashes quite rapidly with the following error:
Steps to reproduce:
Please include a minimal but complete recreation of the problem,
including server configuration, agent(s) used, etc. The easier you make it
for us to reproduce it, the more likely that somebody will take the time to
look at it.
go buildthe project (already built binary might as well have the bug)./apm-server -c <configuration file>, the configuration file must point to a working Elasticsearch node / clusterMy apm-server.yml file looks like:
Provide logs (if relevant):
Crash logs
APM logs
GDB backtrace
** Additional information **
This issue started to happen as soon as we switched to glibc 2.34. It did not happen on glibc 2.33. Downgrading glibc from 2.34 to 2.33 resolves the issue, with the exact same binaries. This problem might as well be a golang bug but I'm not really sure and since I don't see any issue on their bug tracker, I'm wondering if we are the only ones hitting this.
Glibc 2.34 started to use the new
clone3syscall on thread creation (in thepthread_create()function from my understanding). This syscall sometimes returnsEPERMleading to the following crash (https://github.com/golang/go/blob/4e308d73ba3610838305997b6f4793c4f4dcfc4e/src/runtime/cgo/gcc_libinit.c#L94). I say "sometimes" because there are other successfulclone3()calls before.I've seen a few projects that also had issue with this new syscall (Docker, Firefox and Chromium). They all mention issues with their sandbox which is based on seccomp. I see the APM project also depend on https://github.com/elastic/go-seccomp-bpf. I couldn't see it really used or even any syscall filtering on the go-seccomp-bpf project as there are on the previously mentionned projets, but maybe I'm overlooking something?
It's also worth to mention that running as root doesn't fix the
Operation not permittederror.Let me know if you need any other information, I'm not a go expert at all. Thanks for any help you could provide, I'll keep looking on my side.