pybind/mgr: reopen database handle on blocklist by batrick · Pull Request #50291 · ceph/ceph

batrick · 2023-02-27T20:04:55Z

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

batrick · 2023-02-28T16:28:05Z

jenkins test api

batrick · 2023-02-28T20:24:24Z

Waiting for #50065 dependency to be merged.

ajarr

We're trying to solve a similar problem in the rbd_support module where the RADOS client used by the module gets blocklisted. And we want the module to recover without manual intervention, i.e., without requiring the admin to execute ceph mgr fail.

Initially, I tried to implement something similar to what you've done here. When blocklist exception is raised by the module's client, shutdown the the blocklisted client connection, unregister the client, initiate a new connection and register the new client. Then I realized I may not be handling the scenario where mgr fail is executed. When mgr fail is executed, initially the MgrMonitor blocklists the registered clients of the failed mgr, then the failed mgr respawns and the standby mgr becomes active based on the MgrMap updates. Since the client blocklisting happens before the failed mgr respawns, wouldn't the failed mgr modules' rados clients (DB's client in your case) recover undesirtably? The recovered client may not get blocklisted in case the mgr respawns before getting registered. Would this PR face the same problem?

Since we expect the rbd_support module's client to get blocklisted in rare cases, we're thinking of having the rbd_support module indicate to the monitors/mgr that mgr needs to respawn. The mgr respawn already happens now when the list of mgr modules changes. So you can think of this solution as rbd_support module being disabled and enabled when it's RADOS client gets blocklisted. @batrick what do you think?

ajarr · 2023-02-28T21:47:38Z

src/libcephsqlite.cc

+    _disconnect();
+    return _connect();
+  }
+  int unblocklist(rsptr _cluster) {


maybe rename this as reconnect() ? I find this more intuitive than unblocklist. I expected unblocklist to remove the client from the list of blocklisted addresses. The reconnect() member function defined above is never used elsewhere.

Sure, I'll make that change in a new spin of the commits.

Did you missed incorporating the suggested change https://github.com/ceph/ceph/pull/50291/files#r1120824761?

Yes, I'm going with maybe_reconnect as reconnect implies unconditional reconnect.

batrick · 2023-03-01T13:52:17Z

Since the client blocklisting happens before the failed mgr respawns, wouldn't the failed mgr modules' rados clients (DB's client in your case) recover undesirtably? The recovered client may not get blocklisted in case the mgr respawns before getting registered. Would this PR face the same problem?

Yes, that's a potential problem. I think the only way to address that is to have the register_client method block until it sees a new MgrMap with the new addrvec.

I think disabling/enabling the module is too heavy-weight. It should be possible to recover without any race conditions if we make register_client block on the new MgrMap.

ajarr · 2023-03-02T01:22:27Z

Since the client blocklisting happens before the failed mgr respawns, wouldn't the failed mgr modules' rados clients (DB's client in your case) recover undesirtably? The recovered client may not get blocklisted in case the mgr respawns before getting registered. Would this PR face the same problem?

Yes, that's a potential problem. I think the only way to address that is to have the register_client method block until it sees a new MgrMap with the new addrvec.

Whose "new addrvec" are you referring to? I don't quite follow how this would fix the issue. Can you please explain?

Maybe introducing a time delay before a mgr_module's or DB's rados client tries to reconnect might help? The time delay should be longer than the time taken for the failed mgr to respawn.

I think disabling/enabling the module is too heavy-weight. It should be possible to recover without any race conditions if we make register_client block on the new MgrMap.

batrick · 2023-03-02T15:56:31Z

Since the client blocklisting happens before the failed mgr respawns, wouldn't the failed mgr modules' rados clients (DB's client in your case) recover undesirtably? The recovered client may not get blocklisted in case the mgr respawns before getting registered. Would this PR face the same problem?

Yes, that's a potential problem. I think the only way to address that is to have the register_client method block until it sees a new MgrMap with the new addrvec.

Whose "new addrvec" are you referring to? I don't quite follow how this would fix the issue. Can you please explain?

Once creating a RADOS client, the next thing the module/mgr should do is register it in the MgrMap so it can be blocklisted, yes? So block completion of register_client until the mgr receives an updated MgrMap that includes the new registered address. That should resolve any races where the module tries to manipulate data.

Maybe introducing a time delay before a mgr_module's or DB's rados client tries to reconnect might help? The time delay should be longer than the time taken for the failed mgr to respawn.

No, that's racy tool. Just wait for an updated MgrMap. We do this all over Ceph, including the MDS. e.g. look at wait_for_mdsmap in the MDS.

ajarr · 2023-03-02T21:16:26Z

Since the client blocklisting happens before the failed mgr respawns, wouldn't the failed mgr modules' rados clients (DB's client in your case) recover undesirtably? The recovered client may not get blocklisted in case the mgr respawns before getting registered. Would this PR face the same problem?

Yes, that's a potential problem. I think the only way to address that is to have the register_client method block until it sees a new MgrMap with the new addrvec.

Whose "new addrvec" are you referring to? I don't quite follow how this would fix the issue. Can you please explain?

Once creating a RADOS client, the next thing the module/mgr should do is register it in the MgrMap so it can be blocklisted, yes? So block completion of register_client until the mgr receives an updated MgrMap that includes the new registered address. That should resolve any races where the module tries to manipulate data.

When mgr fail is executed, first the registered clients get blocklisted, and then the MgrMap changes are proposed where the standby mgr is set as the active. Upon seeing the MgrMap changes, the standby daemon becomes active, and the failed active mgr respawns. Even with your suggestion, is it possible that the failed mgr-module's blocklisted client reconnects and registers itself before the pending MgrMap changes, where the standby mgr is set as active, are proposed?

In addition to your suggestion that we wait for the client registration to be reflected in the MgrMap, I think we need to blocklist the registered clients when a mgr starts up (Mgr::init) instead of when MgrMonitor is trying to drop the active mgr ( MgrMonitor::drop_active).

Maybe introducing a time delay before a mgr_module's or DB's rados client tries to reconnect might help? The time delay should be longer than the time taken for the failed mgr to respawn.

No, that's racy tool. Just wait for an updated MgrMap. We do this all over Ceph, including the MDS. e.g. look at wait_for_mdsmap in the MDS.

batrick · 2023-03-03T14:16:57Z

Even with your suggestion, is it possible that the failed mgr-module's blocklisted client reconnects and registers itself before the pending MgrMap changes, where the standby mgr is set as active, are proposed?

It can reconnect but it will block on the new MgrMap which includes its addrvec. It's not a problem that the mgr creates a new "untracked" RADOS client if that client has not done anything yet.

In addition to your suggestion that we wait for the client registration to be reflected in the MgrMap, I think we need to blocklist the registered clients when a mgr starts up (Mgr::init) instead of when MgrMonitor is trying to drop the active mgr ( MgrMonitor::drop_active).

I don't see why.

ajarr · 2023-03-05T14:19:58Z

Even with your suggestion, is it possible that the failed mgr-module's blocklisted client reconnects and registers itself before the pending MgrMap changes, where the standby mgr is set as active, are proposed?

It can reconnect but it will block on the new MgrMap which includes its addrvec. It's not a problem that the mgr creates a new "untracked" RADOS client if that client has not done anything yet.

In addition to your suggestion that we wait for the client registration to be reflected in the MgrMap, I think we need to blocklist the registered clients when a mgr starts up (Mgr::init) instead of when MgrMonitor is trying to drop the active mgr ( MgrMonitor::drop_active).

I don't see why.

Can the following happen (see below), where the MgrMap changes to register the reconnected client are proposed before the MgrMap changes to promote standby as active and drop standby?

              MgrMonitor                           Active Mgr to be failed                        Standby mgr to be promoted
--------------------------------------            --------------------------                      ---------------------------     

1. registered clients get blocklisted
                                              2. mgr module's RADOS client reconnects
                                                 to recover from block listing
                                              3. mgr module's RADOS client tries to
                                                 register new client
4. MgrMap changes to register reconnected
   client's address proposed                                           
                                              5. mgr module's RADOS client registration
                                                 complete upon seeing its address in the
                                                 MgrMap
6. MgrMap changes to promote the standby mgr  6. mgr module's RADOS client starts handling
   as active and drop the active mgr are         requests
   proposed.
                                                                                                7. Standby mgr sees the MgrMap changes, becomes active
                                                                                                   and starts loading python modules.

                                              8. mgr module's RADOS client is running              The newly loaded module's RADOS client
                                                                                                   is also running after registering.
                                              9. mgr sees the MgrMap change and respawns,
                                                 which cleans up the RADOS client

batrick · 2023-03-05T20:53:10Z

The "race" you've identified between 6 and 7 has always existed. Your timeline omits that the MgrMonitor in 6 also blocklists the "new" RADOS client created in 2. So the OSDs should stop talking to the old mgr in ~7 (of course, this is a distributed system so not all entities receive maps simultaneously).

The fact that the new active can start doing things before the old instance is completely blocklisted by all OSDs is not something we can feasibly avoid really. We have strategies for scoping the issue:

In CephFS, the new MDS won't do anything until it sees the old instance is blocklisted in the OSDMap (via the last_failure_osd_epoch). That way the MDS can communicate the latest OSDMap epoch when talking to OSDs to ensure that, for the OSDs it does talk to, the old MDS cannot be simultaneously doing things too since it's blocklisted.
In libcephsqlite, the database lock resides on one object. No client of libcephsqlite will use the database without that lock. So, the single OSD that receives the new OSDMap with the blocklisted address is sufficient for preventing erroneous access. (As long as libcephsqlite waits for the grace period of the lock to expire without breaking it, which it does.)

ajarr · 2023-03-06T15:55:38Z

Your timeline omits that the MgrMonitor in 6 also blocklists the "new" RADOS client created in 2.

This is the race I thought I could happen: where in MgrMonitor::prepare_command, in between the failed mgr's client getting blocklisted and the MgrMap changes with standby mgr being set as active being proposed, the failed mgr's client reconnects from being blocklisted and gets itself registered in the MgrMap by MgrMonitor::prepare_beacon.

So the OSDs should stop talking to the old mgr in ~7 (of course, this is a distributed system so not all entities receive maps simultaneously).

The fact that the new active can start doing things before the old instance is completely blocklisted by all OSDs is not something we can feasibly avoid really.

Got it. Thanks for the explanation below.

We have strategies for scoping the issue:

In CephFS, the new MDS won't do anything until it sees the old instance is blocklisted in the OSDMap (via the last_failure_osd_epoch). That way the MDS can communicate the latest OSDMap epoch when talking to OSDs to ensure that, for the OSDs it does talk to, the old MDS cannot be simultaneously doing things too since it's blocklisted.

In libcephsqlite, the database lock resides on one object. No client of libcephsqlite will use the database without that lock. So, the single OSD that receives the new OSDMap with the blocklisted address is sufficient for preventing erroneous access. (As long as libcephsqlite waits for the grace period of the lock to expire without breaking it, which it does.)

batrick · 2023-03-06T16:03:56Z

Your timeline omits that the MgrMonitor in 6 also blocklists the "new" RADOS client created in 2.

This is the race I thought I could happen: where in MgrMonitor::prepare_command, in between the failed mgr's client getting blocklisted and the MgrMap changes with standby mgr being set as active being proposed, the failed mgr's client reconnects from being blocklisted and gets itself registered in the MgrMap by MgrMonitor::prepare_beacon.

It won't be able to register the new RADOS client as the mgr no longer exists in the (pending or not) MgrMap. So the old mgr should never complete the "register_client" call (with the proposed change to have it block on the new MgrMap).

Something that should be done but not, related to your concern, is to "plug" PAXOS whenever we are manipulating the OSDMap (blocklisting) and the MgrMap (failover). You may write up a ticket and trivial fix for that. I don't think it's a significant issue but it would be appropriate.

batrick · 2023-03-06T18:25:03Z

Something that should be done but not, related to your concern, is to "plug" PAXOS whenever we are manipulating the OSDMap (blocklisting) and the MgrMap (failover). You may write up a ticket and trivial fix for that. I don't think it's a significant issue but it would be appropriate.

https://tracker.ceph.com/issues/58923

batrick · 2023-03-06T18:44:40Z

https://tracker.ceph.com/issues/58924

ajarr · 2023-04-21T19:14:16Z

src/mgr/PyModuleRegistry.h

-    clients.emplace(std::string(name), std::move(addrs));
+    auto n = std::string(name);
+    if (replace) {
+      clients.erase(n);


Since multimap clients can have identical keys (client name), I don't think this interface is safe until we make the change suggested here #51169 (comment) . So we drop the "replace" functionality for now?

I think the intention after #51169 is merged is that register_client will always replace?

#51169 will not include the changes to make the data structure of clients from multimap to map. Do you feel that if even if the data type of clients is not changed by #51169, this interface is still safe to have?

I confess I don't like the "replace" interface but I don't see what about it is unsafe.

The "replace" interface can also be removed when/if we switch from multimap to map. It's an internal API so it's not a big deal.

ajarr · 2023-06-06T19:04:19Z

src/libcephsqlite.cc

+    _disconnect();
+    return _connect();
+  }
+  int unblocklist(rsptr _cluster) {


Did you missed incorporating the suggested change https://github.com/ceph/ceph/pull/50291/files#r1120824761?

ajarr · 2023-06-06T19:11:50Z

src/pybind/mgr/mgr_module.py

+            (addrv,) = cur.fetchone()
+            assert addrv is not None
+            self.log.debug(f"new libcephsqlite addrv = {addrv}")
+            self._ceph_register_client("libcephsqlite", addrv, True)


Since the replace functionality of _ceph_register_client() is not safe, just deregister and register the new address?

src/pybind/mgr/mgr_module.py

ajarr

The changes look good to me. My remaining question, #50291 (comment)

@mchangir since you worked on libcephsqlite, can you go through this PR as well?

ajarr · 2023-06-12T12:46:24Z

src/mgr/PyModuleRegistry.h

-    clients.emplace(std::string(name), std::move(addrs));
+    auto n = std::string(name);
+    if (replace) {
+      clients.erase(n);


#51169 will not include the changes to make the data structure of clients from multimap to map. Do you feel that if even if the data type of clients is not changed by #51169, this interface is still safe to have?

ajarr

I see this in the make check failure ,

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/__init__.py", line 60, in <module>
    from .module import Module, StandbyModule  # noqa: F401
  File "/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/module.py", line 24, in <module>
    from mgr_module import CLIReadCommand, CLIWriteCommand, HandleCommandResult, \
  File "/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/mgr_module.py", line 518, in <module>
    def MgrModuleRecoverDB(func: abc.Callable) -> abc.Callable:
NameError: name 'abc' is not defined. Did you mean: 'abs'?

batrick · 2023-06-13T18:18:43Z

I see this in the make check failure ,

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/__init__.py", line 60, in <module>
    from .module import Module, StandbyModule  # noqa: F401
  File "/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/module.py", line 24, in <module>
    from mgr_module import CLIReadCommand, CLIWriteCommand, HandleCommandResult, \
  File "/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/mgr_module.py", line 518, in <module>
    def MgrModuleRecoverDB(func: abc.Callable) -> abc.Callable:
NameError: name 'abc' is not defined. Did you mean: 'abs'?

.. weird I didn't notice that before. I removed abc..

ajarr

@mchangir can you take a look?

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Presently the libcephsqlite library becomes unusable (by design, at the time) when the RADOS instance it uses is blocklisted. Unfortunately, it's fairly common for this to happen during disruptive events. For the ceph-mgr, it's unacceptable to require manual intervention via the `ceph mgr fail` command. So, this commit reworks libcephsqlite to reconnect to RADOS when its cluster handle is blocklisted. This is safe as any open "file" by sqlite3 has its own ptr to the open cluster handle. So those open files will all continually fail until sqlite3 closes the handle (on database connection shutdown). The application can then reopen the database if desired using the fresh RADOS handle. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Two new parameters: - "name" to allow setting the name associated with the addrv. Previously this was hard-coded to the mgr module name. However, we want mgr modules to be able to update the libcephsqlite cluster handle. - "replace" to replace all addrv for the given "name". libcephsqlite only ever has one handle, remove the old ones. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Fixes: https://tracker.ceph.com/issues/55606 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

batrick · 2023-06-27T18:19:25Z

rebase to fix conflict

batrick · 2023-07-12T13:41:29Z

This is ready to merge when RADOS QA is complete. cc @ljflores @rzarzynski

yuriw · 2023-07-14T20:28:19Z

From @kamoltat ref: https://trello.com/c/7y6uj4bo

RADOS approved, all failures/dead jobs are unrelated and known failures.

Failures:

7332480 -> Bug #58946: cephadm: KeyError: 'osdspec_affinity' - Dashboard - Ceph -> cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard
7332558 -> Bug #57302: ERROR: test_get_status (tasks.mgr.dashboard.test_cluster.ClusterTest) - Dashboard - Ceph -> Test failure: test_create_access_permissions (tasks.mgr.dashboard.test_pool.PoolTest)
7332565 -> Bug #57754: test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Infrastructure - Ceph -> test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Infrastructure
7332612 -> Bug #57754: test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Infrastructure - Ceph -> test_envlibrados_for_rocksdb.sh: update-alternatives: error: alternative path /usr/bin/gcc-11 doesn't exist - Infrastructure
7332613 -> Bug #55347: SELinux Denials during cephadm/workunits/test_cephadm - Infrastructure - Ceph -> SELinux Denials during cephadm/workunits/test_cephadm
7332636 -> Bug #58946: cephadm: KeyError: 'osdspec_affinity' - Dashboard - Ceph -> cephadm: KeyError: 'osdspec_affinity' - Ceph - Mgr - Dashboard

Deads:

7332357 -> Bug #61164: Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds - Infrastructure - Ceph -> Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds
7332405 -> Bug #59380: rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - rgw - Ceph -> rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)"
7332559 -> Bug #59380: rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)" - rgw - Ceph -> rados/singleton-nomsgr: test failing from "Health check failed: 1 full osd(s) (OSD_FULL)" and "Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)"

batrick added mgr libcephsqlite ceph SQLite3 VFS labels Feb 27, 2023

github-actions bot added core pybind labels Feb 27, 2023

batrick force-pushed the i55606 branch 3 times, most recently from 1ec5311 to b563a6d Compare February 28, 2023 13:34

batrick mentioned this pull request Feb 28, 2023

mgr: store names of modules that register RADOS clients in the MgrMap #50065

Merged

14 tasks

batrick force-pushed the i55606 branch from b563a6d to 865e8c1 Compare February 28, 2023 20:23

github-actions bot added documentation mon tests labels Feb 28, 2023

batrick force-pushed the i55606 branch from 865e8c1 to f682f88 Compare February 28, 2023 20:24

ajarr reviewed Feb 28, 2023

View reviewed changes

batrick force-pushed the i55606 branch from f682f88 to 67110fa Compare March 9, 2023 19:19

batrick marked this pull request as ready for review March 9, 2023 19:20

batrick requested a review from a team as a code owner March 9, 2023 19:20

batrick force-pushed the i55606 branch from 67110fa to ac58f18 Compare May 10, 2023 15:40

github-actions bot removed the needs-rebase label May 10, 2023

batrick force-pushed the i55606 branch from ac58f18 to b9ede86 Compare May 15, 2023 17:13

ajarr requested changes Jun 6, 2023

View reviewed changes

ajarr reviewed Jun 6, 2023

View reviewed changes

src/pybind/mgr/mgr_module.py Show resolved Hide resolved

batrick force-pushed the i55606 branch from b9ede86 to 8b6a114 Compare June 10, 2023 14:55

ajarr approved these changes Jun 12, 2023

View reviewed changes

ajarr requested a review from mchangir June 12, 2023 12:50

ajarr requested changes Jun 13, 2023

View reviewed changes

batrick force-pushed the i55606 branch from 8b6a114 to 6db9ead Compare June 13, 2023 18:17

ajarr approved these changes Jun 15, 2023

View reviewed changes

batrick added needs-qa wip-pdonnell-testing and removed wip-pdonnell-testing labels Jun 22, 2023

batrick added 6 commits June 27, 2023 14:18

pybind/mgr: remove unused variable

dd901eb

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

SimpleRADOSStriper: increase debugging level for critical errors

8f073b1

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr: reopen database handle on blocklist

bc40217

Fixes: https://tracker.ceph.com/issues/55606 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add test case for libcephsqlite maybe_reconnect

c010e8a

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

batrick force-pushed the i55606 branch from 6db9ead to c010e8a Compare June 27, 2023 18:19

ljflores added the wip-yuri6-testing label Jul 10, 2023

yuriw merged commit deae3a6 into ceph:main Jul 14, 2023

batrick deleted the i55606 branch July 14, 2023 20:29

This was referenced Jul 14, 2023

reef: pybind/mgr: reopen database handle on blocklist #52460

Merged

quincy: pybind/mgr: reopen database handle on blocklist #52461

Merged

Conversation

batrick commented Feb 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

batrick commented Feb 28, 2023

Uh oh!

batrick commented Feb 28, 2023

Uh oh!

ajarr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajarr Jun 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

batrick commented Mar 1, 2023

Uh oh!

ajarr commented Mar 2, 2023

Uh oh!

batrick commented Mar 2, 2023

Uh oh!

ajarr commented Mar 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

batrick commented Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ajarr commented Mar 5, 2023

Uh oh!

batrick commented Mar 5, 2023

Uh oh!

ajarr commented Mar 6, 2023

Uh oh!

batrick commented Mar 6, 2023

Uh oh!

batrick commented Mar 6, 2023

Uh oh!

batrick commented Mar 6, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajarr Jun 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ajarr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajarr left a comment

Choose a reason for hiding this comment

Uh oh!

batrick commented Jun 13, 2023

Uh oh!

ajarr left a comment

Choose a reason for hiding this comment

Uh oh!

batrick commented Jun 27, 2023

batrick commented Feb 27, 2023 •

edited

Loading

ajarr left a comment •

edited

Loading

ajarr Jun 6, 2023 •

edited

Loading

ajarr commented Mar 2, 2023 •

edited

Loading

batrick commented Mar 3, 2023 •

edited

Loading

ajarr Jun 6, 2023 •

edited

Loading

ajarr left a comment •

edited

Loading