Prometheus Agent crashes after upgrading to v3.9.0 #17800
Description
What did you do?
After upgrading prometheus (deployed via docker in agent mode) to v3.9.0 it crashes very shortly after startup.
The image used is docker.io/prom/prometheus:v3.9.0.
The arguments are
"Entrypoint": [
"/bin/prometheus"
],
"Cmd": [
"--agent",
"--config.file=/etc/prometheus/prometheus.yaml",
"--storage.agent.path=/prometheus",
"--web.console.libraries=/usr/share/prometheus/console_libraries",
"--web.console.templates=/usr/share/prometheus/consoles",
"--web.external-url=https://prom01.example.com/",
"--web.enable-lifecycle"
],What did you see instead? Under which circumstances?
Prometheus crashes shortly after startup.
System information
Linux 5.14.0-570.58.1.el9_6.x86_64
Prometheus version
prometheus, version 3.9.0 (branch: HEAD, revision: cd875bd8c9211d7606981223d59ab3adf73432f2)
build user: root@b4c90e3600b7
build date: 20260106-16:54:35
go version: go1.25.5
platform: linux/amd64
tags: netgo,builtinassets
Prometheus configuration file
global:
external_labels: {prometheus: default, prometheus_replica: prom01}
remote_write:
- name: remotewritereceiver
url: "https://rw.example.com/api/v1/write"
write_relabel_configs:
- regex: ([a-z0-9_-]+)/([a-z0-9_-]+)
replacement: $1
source_labels: [job]
target_label: __tenant_id__Logs
time=2026-01-07T07:25:35.887Z level=INFO source=main.go:1589 msg="updated GOGC" old=100 new=75
time=2026-01-07T07:25:35.887Z level=INFO source=main.go:704 msg="Leaving GOMAXPROCS=8: CPU quota undefined" component=automaxprocs
time=2026-01-07T07:25:35.887Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14816869171 previous=9223372036854775807
time=2026-01-07T07:25:35.887Z level=INFO source=main.go:803 msg="Starting Prometheus Agent" mode=agent version="(version=3.9.0, branch=HEAD, revision=cd875bd8c9211d7606981223d59ab3adf73432f2)"
time=2026-01-07T07:25:35.887Z level=INFO source=main.go:808 msg="operational information" build_context="(go=go1.25.5, platform=linux/amd64, user=root@b4c90e3600b7, date=20260106-16:54:35, tags=netgo,builtinassets)" host_details="(Linux 5.14.0-570.58.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Oct 21 04:15:07 EDT 2025 x86_64 dd3acc1b75cb (none))" fd_limits="(soft=1073741816, hard=1073741816)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-01-07T07:25:35.889Z level=INFO source=web.go:684 msg="Start listening for connections" component=web address=0.0.0.0:9090
time=2026-01-07T07:25:35.889Z level=INFO source=main.go:1388 msg="Starting WAL storage ..."
time=2026-01-07T07:25:35.890Z level=WARN source=dir_locker.go:76 msg="A lockfile from a previous execution already existed. It was replaced" file=/prometheus/lock
time=2026-01-07T07:25:35.892Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=[::]:9090
time=2026-01-07T07:25:35.892Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=[::]:9090
time=2026-01-07T07:25:35.894Z level=INFO source=db.go:393 msg="replaying WAL, this may take a while" dir=/prometheus/wal
time=2026-01-07T07:25:35.900Z level=INFO source=db.go:421 msg="WAL checkpoint loaded"
time=2026-01-07T07:25:35.938Z level=INFO source=db.go:445 msg="WAL segment loaded" segment=5029 maxSegment=5158
[...........]
time=2026-01-07T07:25:36.069Z level=INFO source=db.go:445 msg="WAL segment loaded" segment=5158 maxSegment=5158
time=2026-01-07T07:25:36.069Z level=INFO source=main.go:1409 msg=XFS_SUPER_MAGIC
time=2026-01-07T07:25:36.069Z level=INFO source=main.go:1412 msg="Agent WAL storage started"
time=2026-01-07T07:25:36.069Z level=INFO source=main.go:1542 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yaml
time=2026-01-07T07:25:36.070Z level=INFO source=watcher.go:240 msg="Starting WAL watcher" component=remote remote_name=remotewritereceiver url=https://rw.example.com/api/v1/write queue=remotewritereceiver
time=2026-01-07T07:25:36.070Z level=INFO source=metadata_watcher.go:90 msg="Starting scraped metadata watcher" component=remote remote_name=remotewritereceiver url=https://rw.example.com/api/v1/write
time=2026-01-07T07:25:36.070Z level=INFO source=watcher.go:292 msg="Replaying WAL" component=remote remote_name=remotewritereceiver url=https://rw.example.com/api/v1/write queue=remotewritereceiver
time=2026-01-07T07:25:36.071Z level=INFO source=main.go:1582 msg="Completed loading of configuration file" db_storage=605ns remote_storage=766.229µs web_handler=746ns query_engine=217ns scrape=793.982µs scrape_sd=562.461µs notify=1.431µs notify_sd=928ns rules=317ns tracing=4.056µs filename=/etc/prometheus/prometheus.yaml totalDuration=2.463554ms
time=2026-01-07T07:25:36.071Z level=INFO source=main.go:1316 msg="Server is ready to receive web requests."
time=2026-01-07T07:25:46.301Z level=INFO source=watcher.go:538 msg="Done replaying WAL" component=remote remote_name=remotewritereceiver url=https://rw.example.com/api/v1/write duration=10.230926207s
panic: interface conversion: *agent.appenderBase is not storage.Appender: missing method Append
goroutine 542 [running]:
github.com/prometheus/prometheus/tsdb/agent.(*DB).Appender(0xc0004c83c0?, {0xc000daba60?, 0xb57f19?})
/app/tsdb/agent/db.go:788 +0x5c
main.(*readyStorage).Appender(0x0?, {0x62099c8, 0xc0006585a0})
/app/cmd/prometheus/main.go:1742 +0x38
github.com/prometheus/prometheus/storage.(*fanout).Appender(0xc0005769f0, {0x62099c8, 0xc0006585a0})
/app/storage/fanout.go:121 +0x42
github.com/prometheus/prometheus/scrape.newScrapePool.func1.3({0x62099c8?, 0xc0006585a0?})
/app/scrape/scrape.go:202 +0x2f
github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport(0xc00063a000, {0xee0f00182?, 0x881a3e0?, 0x0?}, {0xee0f00182?, 0x881a3e0?, 0x881a3e0?}, 0x0)
/app/scrape/scrape.go:1431 +0x397
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc00063a000, 0x0)
/app/scrape/scrape.go:1391 +0x365
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync in goroutine 579
/app/scrape/scrape.go:626 +0xc7d