-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Closed
Description
The current prometheus is crashing alertmanager when sending alerts:
2014/04/03 11:42:04 /go/src/code.google.com/p/gorest/gorest.go:194 (0x572cf3)
google.com/p/gorest.func.001: log.Printf("%s", debug.Stack())
/usr/local/go/src/pkg/runtime/panic.c:248 (0x415766)
panic: runtime·newstackcall(d->fn, (byte*)d->args, d->siz);
/usr/local/go/src/pkg/runtime/panic.c:482 (0x41600d)
panicstring: runtime·panic(err);
/usr/local/go/src/pkg/runtime/os_linux.c:234 (0x414eaa)
sigpanic: runtime·panicstring("invalid memory address or nil pointer dereference");
/go/src/github.com/prometheus/alertmanager/web/api/alert.go:40 (0x46d70c)
com/prometheus/alertmanager/web/api.AlertManagerService.AddAlerts: s.Manager.Receive(as)
/go/src/github.com/prometheus/alertmanager/web/api/alert.go:1 (0x46e6d4)
com/prometheus/alertmanager/web/api.(*AlertManagerService).AddAlerts: // Copyright 2013 Prometheus Team
/usr/local/go/src/pkg/runtime/asm_amd64.s:339 (0x426b32)
call32: CALLFN(call32, 32)
/usr/local/go/src/pkg/reflect/value.go:474 (0x490a7b)
Value.call: call(fn, ptr, uint32(size))
/usr/local/go/src/pkg/reflect/value.go:345 (0x48fb6d)
Value.Call: return v.call("Call", in)
/go/src/code.google.com/p/gorest/reflect.go:391 (0x5708c0)
google.com/p/gorest.prepareServe: ret = servVal.Method(ep.methodNumberInParent).Call(arrArgs)
/go/src/code.google.com/p/gorest/gorest.go:215 (0x56771b)
google.com/p/gorest.manager.ServeHTTP: data, state := prepareServe(ctx, ep)
/go/src/code.google.com/p/gorest/api.go:1 (0x5739ae)
google.com/p/gorest.(*manager).ServeHTTP: //Copyright 2011 Siyabonga Dlamini (siyabonga.dlamini@gmail.com). All rights reserved.
/go/src/github.com/prometheus/alertmanager/web/compression.go:90 (0x46b2b2)
com/prometheus/alertmanager/web.compressionHandler.ServeHTTP: c.handler.ServeHTTP(compWriter, req)
/go/src/github.com/prometheus/alertmanager/web/alerts.go:1 (0x46d3ee)
com/prometheus/alertmanager/web.(*compressionHandler).ServeHTTP: // Copyright 2013 Prometheus Team
/go/src/github.com/prometheus/client_golang/prometheus/exp/coarsemux.go:65 (0x573e0a)
com/prometheus/client_golang/prometheus/exp.handlerDelegator.ServeHTTP: h.delegate.ServeHTTP(rwd, r)
/go/src/github.com/prometheus/client_golang/prometheus/exp/coarsemux.go:1 (0x5755b4)
com/prometheus/client_golang/prometheus/exp.(*handlerDelegator).ServeHTTP: // Copyright (c) 2013, Prometheus Team
/usr/local/go/src/pkg/net/http/server.go:1496 (0x537f03)
(*ServeMux).ServeHTTP: h.ServeHTTP(w, r)
/usr/local/go/src/pkg/net/http/server.go:1597 (0x53870e)
serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
/usr/local/go/src/pkg/net/http/server.go:1167 (0x5365a7)
(*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
/usr/local/go/src/pkg/runtime/proc.c:1394 (0x419a60)
goexit: runtime·goexit(void)
I've did a tcpdump and reproduced what prometheus posts:
curl -L -d '[{"Description":"http://192.168.100.66:9080/metrics has been down for more than 5 minutes.","Labels":{"alertname":"NodeDown","instance":"http://192.168.100.66:9080/metrics","job":"node_exporter","severity":"page"},"Payload":{"ActiveSince":"2014-04-03T11:20:49Z","AlertingRule":"ALERT NodeDown IF (up == 0) FOR 2m WITH {severity=\"page\"}","GeneratorUrl":"http://19275cb8ad55:9090/graph#%5B%7B%22expr%22%3A%22%28up%20%3D%3D%200%29%22%2C%22tab%22%3A1%7D%5D","Value":"0.000000"},"Summary":"Node http://192.168.100.66:9080/metrics down"}]' -X POST localhost:9090/api/alerts
It seems that in web/api/alert.go:40 s.Manager is nil.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels