Actions
Bug #68146
closedNFS Ganesha fsal CEPH fio denali workload causing "ms_dispatch" received signal SIGSEGV, Segmentation fault.
Status:
Resolved
Priority:
High
Assignee:
Category:
Correctness/Safety
Target version:
% Done:
0%
Source:
Community (user)
Backport:
squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Client, Ganesha FSAL
Labels (FS):
Pull request ID:
Tags (freeform):
backport_processed
Merge Commit:
Fixed In:
v19.3.0-5458-g43012083f4
Released In:
v20.2.0~1832
Upkeep Timestamp:
2025-11-01T01:38:56+00:00
Description
on ceph main branch and ganesha branch:
❯ git branch -vv * main 08cec6974d8 [origin/main: behind 56] Merge pull request #58728 from ifed01/wip-ifed-ret-error-kv-stats ❯ git branch -vv * (HEAD detached at V6-dev.19) ced8e8578 V6-dev.19
executing the fio workload:
sudo numactl -N 1 -m 1 -- fio --ioengine=libaio -directory=/mnt/NFS --filename=fio.test --name=job --bs=16k --direct=1 --group_reporting --iodepth=128 --randrepeat=0 --norandommap=1 --thread=1 --ramp_time=20s --offset_increment=5% --size=2G --time_based --runtime=60 --ramp_time=1s --percentage_random=0 --rw=write --rwmixread=50
results consistently with the segf in the ganesha.nfsd process:
Thread 53 "ms_dispatch" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffe01f2640 (LWP 1074204)] 0x0000011885030043 in ?? () Missing separate debuginfos, use: dnf debuginfo-install glibc-2.34-100.el9_4.3.x86_64 gperftools-libs-2.9.1-2.el9.x86_64 gssproxy-0.8.4-6.el9.x86_64 keyutils-libs-1.6.3-1.el9.x86_64 krb5-libs-1.21.1-1.el9.x86_64 libacl-2.3.1-4.el9.x86_64 libattr-2.5.1-3.el9.x86_64 libblkid-2.37.4-18.el9.x86_64 libcom_err-1.46.5-5.el9.x86_64 libibverbs-46.0-1.el9.x86_64 libnl3-3.7.0-1.el9.x86_64 libnsl2-2.0.0-1.el9.x86_64 librdmacm-46.0-1.el9.x86_64 libselinux-3.6-1.el9.x86_64 libstdc++-11.4.1-3.el9.x86_64 libtirpc-1.3.3-8.el9_4.x86_64 libunwind-1.6.2-1.el9.x86_64 libuuid-2.37.4-18.el9.x86_64 libwbclient-4.19.4-105.el9_4.x86_64 openssl-libs-3.0.7-24.el9.x86_64 systemd-libs-252-32.el9_4.x86_64 userspace-rcu-0.12.1-6.el9.x86_64 zlib-1.2.11-40.el9.x86_64 (gdb) bt #0 0x0000011885030043 in ?? () #1 0x00007fffe6b65f9a in finish_contexts<std::vector<Context*, std::allocator<Context*> > > (cct=0x1346000, finished=..., result=0) at ../src/include/Context.h:209 #2 0x00007fffe6abddcf in Client::signal_context_list (this=0x1b2a800, ls=<error reading variable: Cannot access memory at address 0x8>) at ../src/client/Client.h:1062 #3 Client::signal_caps_inode (this=0x1b2a800, in=0x1526600) at ../src/client/Client.cc:4282 #4 Client::handle_cap_flush_ack (this=0x1b2a800, session=0xc2d1e0, in=0x1526600, cap=<optimized out>, m=...) at ../src/client/Client.cc:5623 #5 0x00007fffe6a9d897 in Client::handle_caps (this=0x1b2a800, m=...) at ../src/client/Client.cc:5451 #6 0x00007fffe6a979d4 in Client::ms_dispatch2 (this=0x1b2a800, m=...) at ../src/client/Client.cc:2994 #7 0x00007ffff5cdc25d in Messenger::ms_deliver_dispatch (this=0xd25200, m=...) at ../src/msg/Messenger.h:740 #8 0x00007ffff5cdb223 in DispatchQueue::entry (this=0xd25508) at ../src/msg/DispatchQueue.cc:201 #9 0x00007ffff5e1e1a9 in DispatchQueue::DispatchThread::entry (this=<optimized out>) at ../src/msg/DispatchQueue.h:101 #10 0x00007ffff5a2612c in Thread::entry_wrapper (this=0xd25658) at ../src/common/Thread.cc:87 #11 0x00007ffff7489c02 in start_thread () from /lib64/libc.so.6 #12 0x00007ffff750ec40 in clone3 () from /lib64/libc.so.6 (gdb)
Updated by Venky Shankar over 1 year ago
- Category set to Correctness/Safety
- Assignee set to Kotresh Hiremath Ravishankar
- Target version set to v20.0.0
- Source set to Community (user)
- Backport set to quincy,reef,squid
Kotresh, please take this one.
Updated by Venky Shankar over 1 year ago
- Status changed from New to Triaged
- Component(FS) Client added
Updated by Venky Shankar over 1 year ago
This seems to be realted to one of the async completion callback running without client_lock held.
Updated by Kotresh Hiremath Ravishankar over 1 year ago
I tried with the following versions and hit the issue (rather a different assert) but it seems the underlying issue is the same.
ceph - commit bf04ea74064910371dea7b9e873202a7b10f2776 (origin/main) nfs-ganesha - Ganesha Version V6-dev.19-0-gced8e8578
Testcase:
1. Start the vstart cluster with nfs 2. sudo mount -t nfs -o port=44545 192.168.1.114:/cephfs /home/kotresh/ganesh_mount1 3. touch /home/kotresh/ganesh_mount1/file1 4. echo 'aaaa' >> /home/kotresh/ganesh_mount1/file1 - Hits the assert in asyn io path
Here is the assert hit 'clnt->client_lock).is_locked_by_me()' So essentially as Venky guessed, client_lock should have been taken but the lock is not held. Since this is single client, single file append use case, the assert is hit with out corruption.
If there are multiple file io (as in fio), with out the client_lock, definitely this could hit corruption. I think which is what is seen in this tracker.
#0 0x00007faa6820d664 in __pthread_kill_implementation () from /lib64/libc.so.6
#1 0x00007faa681b4c4e in raise () from /lib64/libc.so.6
#2 0x00007faa6819c902 in abort () from /lib64/libc.so.6
#3 0x00007faa66e0ac45 in ceph::__ceph_assert_fail (assertion=0x7faa5c7a9d30 "((clnt->client_lock).is_locked_by_me())", file=<optimized out>, line=11869, func=0x7faa5c7a9dc8 "void Client::C_nonblocking_fsync_state::advance()") at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/common/assert.cc:75
#4 0x00007faa66e0ad20 in ceph::__ceph_assert_fail (ctx=...) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/common/assert.cc:80
#5 0x00007faa5c691804 in Client::C_nonblocking_fsync_state::advance (this=this@entry=0x7fa930009bb0) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/client/Client.cc:11869
#6 0x00007faa5c6934de in Client::C_Write_Finisher::try_complete (this=this@entry=0x7fa930016ea0) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/client/Client.cc:11463
#7 0x00007faa5c6b498f in Client::C_Write_Finisher::finish_io (this=0x7fa930016ea0, r=5) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/client/Client.cc:11414
#8 0x00007faa5c718889 in Client::CWF_iofinish::finish (this=<optimized out>, r=<optimized out>) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/client/Client.h:1464
#9 0x00007faa5c64b627 in Context::complete (this=0x7fa930008840, r=<optimized out>) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/include/Context.h:99
#10 0x00007faa5c78698d in Objecter::Op::complete(std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&, boost::system::error_code, int, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>)::{lambda(auto:1&&)#1}::operator()<Context*>(Context*&&) const (__closure=0x7faa2bdfe8c0, arg=@0x7faa2bdfeae0: 0x7fa930008840) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/osdc/Objecter.h:2075
#11 std::__invoke_impl<void, Objecter::Op::complete(std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&, boost::system::error_code, int, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>)::{lambda(auto:1&&)#1}, Context*>(std::__invoke_other, Objecter::Op::complete(std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&, boost::system::error_code, int, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>)::{lambda(auto:1&&)#1}&&, Context*&&) (__f=...) at /usr/include/c++/14/bits/invoke.h:61
#12 std::__invoke<Objecter::Op::complete(std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&, boost::system::error_code, int, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>)::{lambda(auto:1&&)#1}, Context*>(Objecter::Op::complete(std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&, boost::system::error_code, int, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>)::{lambda(auto:1&&)#1}&&, Context*&&) (__fn=...) at /usr/include/c++/14/bits/invoke.h:96
#13 std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<void> (*)(Objecter::Op::complete(std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&, boost::system::error_code, int, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>)::{lambda(auto:1&&)#1}&&, std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&)>, std::integer_sequence<unsigned long, 2ul> >::__visit_invoke(Objecter::Op::complete(std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&, boost::system::error_code, int, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>)::{lambda(auto:1&&)#1}&&, std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&) (__visitor=..., __vars#0=...) at /usr/include/c++/14/variant:1060
#14 std::__do_visit<std::__detail::__variant::__deduce_visit_result<void>, Objecter::Op::complete(std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&, boost::system::error_code, int, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>)::{lambda(auto:1&&)#1}, std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*> >(Objecter::Op::complete(std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&, boost::system::error_code, int, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>)::{lambda(auto:1&&)#1}&&, std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&) (__visitor=...) at /usr/include/c++/14/variant:1821
#15 0x00007faa5c7869af in std::visit<Objecter::Op::complete(std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&, boost::system::error_code, int, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>)::{lambda(auto:1&&)#1}, std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*> >(Objecter::Op::complete(std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&, boost::system::error_code, int, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>)::{lambda(auto:1&&)#1}&&, std::variant<boost::asio::any_completion_handler<void (boost::system::error_code)>, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code)> >, Context*>&&) (__visitor=...) at /usr/include/c++/14/variant:1882
#16 0x00007faa5c7869ff in Objecter::Op::complete (f=..., ec=..., r=r@entry=0, e=...) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/osdc/Objecter.h:2072
#17 0x00007faa5c75d6e6 in Objecter::handle_osd_op_reply (this=this@entry=0x365d9e40, m=m@entry=0x7fa9d0009ba0) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/osdc/Objecter.cc:3643
#18 0x00007faa5c75e0ed in Objecter::ms_dispatch (this=0x365d9e40, m=0x7fa9d0009ba0) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/osdc/Objecter.cc:990
#19 0x00007faa5c760af8 in Objecter::ms_fast_dispatch (this=<optimized out>, m=0x7fa9d0009ba0) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/osdc/Objecter.h:2772
#20 0x00007faa5c6cb9a7 in Dispatcher::ms_fast_dispatch2 (this=0x365d9e48, m=...) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/Dispatcher.h:90
#21 0x00007faa66f36e56 in Messenger::ms_fast_dispatch (this=<optimized out>, m=...) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/Messenger.h:713
#22 0x00007faa66f3614c in DispatchQueue::fast_dispatch (this=0x365d5e58, m=...) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/DispatchQueue.cc:74
#23 0x00007faa66fde389 in DispatchQueue::fast_dispatch (this=<optimized out>, m=m@entry=0x7fa9d0009ba0) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/DispatchQueue.h:203
#24 0x00007faa6700f0e3 in ProtocolV2::handle_message (this=this@entry=0x7fa930014a70) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/async/ProtocolV2.cc:1513
#25 0x00007faa6701b91f in ProtocolV2::handle_read_frame_dispatch (this=this@entry=0x7fa930014a70) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/async/ProtocolV2.cc:1170
#26 0x00007faa6701bab4 in ProtocolV2::_handle_read_frame_epilogue_main (this=this@entry=0x7fa930014a70) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/async/ProtocolV2.cc:1360
#27 0x00007faa6701d258 in ProtocolV2::handle_read_frame_epilogue_main (this=0x7fa930014a70, buffer=..., r=0) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/async/ProtocolV2.cc:1337
#28 0x00007faa6701d350 in CtRxNode<ProtocolV2>::call (this=<optimized out>, foo=<optimized out>) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/async/Protocol.h:67
#29 0x00007faa6700b191 in ProtocolV2::run_continuation (this=0x7fa930014a70, continuation=...) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/async/ProtocolV2.cc:49
#30 0x00007faa6700b6ba in operator() (__closure=0x7fa928005328, buffer=<optimized out>, r=0) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/async/ProtocolV2.cc:782
#31 0x00007faa6700b717 in std::__invoke_impl<void, ProtocolV2::read(CONTINUATION_RXBPTR_TYPE<ProtocolV2>&, rx_buffer_t&&)::<lambda(char*, int)>&, char*, long int> (__f=...) at /usr/include/c++/14/bits/invoke.h:61
#32 std::__invoke_r<void, ProtocolV2::read(CONTINUATION_RXBPTR_TYPE<ProtocolV2>&, rx_buffer_t&&)::<lambda(char*, int)>&, char*, long int> (__fn=...) at /usr/include/c++/14/bits/invoke.h:111
#33 std::_Function_handler<void(char*, long int), ProtocolV2::read(CONTINUATION_RXBPTR_TYPE<ProtocolV2>&, rx_buffer_t&&)::<lambda(char*, int)> >::_M_invoke(const std::_Any_data &, char *&&, long &&) (__functor=..., __args#0=<optimized out>, __args#1=<optimized out>) at /usr/include/c++/14/bits/std_function.h:290
#34 0x00007faa66fde8cf in std::function<void(char*, long)>::operator() (this=this@entry=0x7fa928005328, __args#0=<optimized out>, __args#1=<optimized out>, __args#1@entry=0) at /usr/include/c++/14/bits/std_function.h:591
#35 0x00007faa66fdc801 in AsyncConnection::process (this=0x7fa928004f50) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/async/AsyncConnection.cc:478
#36 0x00007faa66fdfa3f in C_handle_read::do_request (this=<optimized out>, fd_or_id=<optimized out>) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/async/AsyncConnection.cc:71
#37 0x00007faa67028a71 in EventCenter::process_events (this=0x36549798, timeout_microseconds=<optimized out>, timeout_microseconds@entry=30000000, working_dur=working_dur@entry=0x7faa2bdff2f8) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/async/Event.cc:436
#38 0x00007faa6702dc55 in operator() (__closure=0x36535e78) at /home/kotresh/sandbox/upstream/kotresh-ceph2/ceph/src/msg/async/Stack.cc:50
#39 0x00007faa6702ddc9 in std::__invoke_impl<void, NetworkStack::add_thread(Worker*)::<lambda()>&> (__f=...) at /usr/include/c++/14/bits/invoke.h:60
#40 std::__invoke_r<void, NetworkStack::add_thread(Worker*)::<lambda()>&> (__fn=...) at /usr/include/c++/14/bits/invoke.h:111
#41 std::_Function_handler<void(), NetworkStack::add_thread(Worker*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/14/bits/std_function.h:290
#42 0x00007faa6702d674 in std::function<void()>::operator() (this=<optimized out>) at /usr/include/c++/14/bits/std_function.h:591
#43 0x00007faa6702d68b in std::__invoke_impl<void, std::function<void ()>>(std::__invoke_other, std::function<void ()>&&) (__f=...) at /usr/include/c++/14/bits/invoke.h:60
#44 std::__invoke<std::function<void ()>>(std::function<void ()>&&) (__fn=...) at /usr/include/c++/14/bits/invoke.h:96
#45 std::thread::_Invoker<std::tuple<std::function<void()> > >::_M_invoke<0ul> (this=<optimized out>) at /usr/include/c++/14/bits/std_thread.h:301
#46 std::thread::_Invoker<std::tuple<std::function<void()> > >::operator() (this=<optimized out>) at /usr/include/c++/14/bits/std_thread.h:308
#47 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::function<void()> > > >::_M_run (this=<optimized out>) at /usr/include/c++/14/bits/std_thread.h:253
#48 0x00007faa662e7564 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#49 0x00007faa6820b6d7 in start_thread () from /lib64/libc.so.6
#50 0x00007faa6828f60c in clone3 () from /lib64/libc.so.6
Updated by Kotresh Hiremath Ravishankar over 1 year ago
- Status changed from Triaged to In Progress
- Pull request ID set to 59987
Updated by Kotresh Hiremath Ravishankar over 1 year ago
- Related to Bug #68308: client: libcephfs aio crashes with assert "cap_refs[c] > 0" added
Updated by Kotresh Hiremath Ravishankar over 1 year ago
- Related to Bug #68309: client: libcephfs aio crashes with assert "ceph_assert(len > 0)"" added
Updated by Venky Shankar over 1 year ago
- Status changed from In Progress to Pending Backport
- Backport changed from quincy,reef,squid to squid
Updated by Upkeep Bot over 1 year ago
- Copied to Backport #68495: squid: NFS Ganesha fsal CEPH fio denali workload causing "ms_dispatch" received signal SIGSEGV, Segmentation fault. added
Updated by Upkeep Bot over 1 year ago
- Tags (freeform) set to backport_processed
Updated by Upkeep Bot 9 months ago
- Merge Commit set to 43012083f476f0a4d9b432fc2b51251d8c3efa8a
- Fixed In set to v19.3.0-5458-g43012083f47
- Upkeep Timestamp set to 2025-07-08T18:44:58+00:00
Updated by Upkeep Bot 8 months ago
- Fixed In changed from v19.3.0-5458-g43012083f47 to v19.3.0-5458-g43012083f476
- Upkeep Timestamp changed from 2025-07-08T18:44:58+00:00 to 2025-07-14T15:45:18+00:00
Updated by Upkeep Bot 8 months ago
- Fixed In changed from v19.3.0-5458-g43012083f476 to v19.3.0-5458-g43012083f4
- Upkeep Timestamp changed from 2025-07-14T15:45:18+00:00 to 2025-07-14T21:09:46+00:00
Updated by Upkeep Bot 6 months ago
- Status changed from Pending Backport to Resolved
- Upkeep Timestamp changed from 2025-07-14T21:09:46+00:00 to 2025-09-23T12:48:35+00:00
Updated by Upkeep Bot 5 months ago
- Released In set to v20.2.0~1832
- Upkeep Timestamp changed from 2025-09-23T12:48:35+00:00 to 2025-11-01T01:38:56+00:00
Actions