Skip to content

debug roachtest: roachtest/manual/monitor/test-failure/node-fatal-explicit-monitor failed #152540

@williamchoe3

Description

@williamchoe3

roachtest.roachtest/manual/monitor/test-failure/node-fatal-explicit-monitor failed with artifacts on release-willdummybranch @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

Failed with:

(monitor.go:267).Wait: monitor failure: dial tcp 127.0.0.1:29000: connect: connection refused
test artifacts and logs in: artifacts/roachtest/manual/monitor/test-failure/node-fatal-explicit-monitor/cpu_arch=arm64/run_1

Fatal entries found in Cockroach logs:

F250826 19:49:07.194443 3106 sql/sem/builtins/builtins.go:6063 ⋮ [T1,Vsystem,n1,client=127.0.0.1:54552,hostssl,user=‹roachprod›] 250  force_log_fatal(): ‹oops›
F250826 19:49:07.194443 3106 sql/sem/builtins/builtins.go:6063 ⋮ [T1,Vsystem,n1,client=127.0.0.1:54552,hostssl,user=‹roachprod›] 250 !goroutine 3106 [running]:
F250826 19:49:07.194443 3106 sql/sem/builtins/builtins.go:6063 ⋮ [T1,Vsystem,n1,client=127.0.0.1:54552,hostssl,user=‹roachprod›] 250 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x1021c7af8?, 0x38?})
F250826 19:49:07.194443 3106 sql/sem/builtins/builtins.go:6063 ⋮ [T1,Vsystem,n1,client=127.0.0.1:54552,hostssl,user=‹roachprod›] 250 !	pkg/util/allstacks/allstacks.go:31 +0x70
F250826 19:49:07.194443 3106 sql/sem/builtins/builtins.go:6063 ⋮ [T1,Vsystem,n1,client=127.0.0.1:54552,hostssl,user=‹roachprod›] 250 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F250826 19:49:07.194443 3106 sql/sem/builtins/builtins.go:6063 ⋮ [T1,Vsystem,n1,client=127.0.0.1:54552,hostssl,user=‹roachprod›] 250 !	pkg/util/allstacks/allstacks.go:18
F250826 19:49:07.194443 3106 sql/sem/builtins/builtins.go:6063 ⋮ [T1,Vsystem,n1,client=127.0.0.1:54552,hostssl,user=‹roachprod›] 250 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x1400201e8d0, 0x24}, {0x1076db81f, 0x1}, {0x1076db81f, 0x1}, {0x1075f672c, 0x6}, {0x1076db81f, ...}}, ...})
F250826 19:49:07.194443 3106 sql/sem/builtins/builtins.go:6063 ⋮ [T1,Vsystem,n1,client=127.0.0.1:54552,hostssl,user=‹roachprod›] 250 !	pkg/util/log/clog.go:294 +0xb0
F250826 19:49:07.194443 3106 sql/sem/builtins/builtins.go:6063 ⋮ [T1,Vsystem,n1,client=127.0.0.1:54552,hostssl,user=‹roachprod›] 250 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x109736b18, 0x1401134ed20}, 0x2, 0x4, 0x0, 0x0?, {0x10765823d, 0x15}, {0x1401148ada8, 0x1, ...})
F250826 19:49:07.194443 3106 sql/sem/builtins/builtins.go:6063 ⋮ [T1,Vsystem,n1,client=127.0.0.1:54552,hostssl,user=‹roachprod›] 250 !	pkg/util/log/channels.go:104 +0x44c

Parameters:

  • arch=arm64
  • cloud=local
  • coverageBuild=false
  • cpu=4
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicBufferedSender=true
  • metamorphicWriteBuffering=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for local clusters

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-53907

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions