tools/cephfs: recover alternate_name of dentries from journal by batrick · Pull Request #55792 · ceph/ceph

batrick · 2024-02-27T18:32:40Z

Right now the cephfs-journal-tool always uses the legacy encoding for dentries which will drop the alternate_name if ever set.

Fixes: https://tracker.ceph.com/issues/64602

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows
jenkins test rook e2e

batrick · 2024-02-27T20:03:07Z

Obviously, I think tests need written for this but unfortunately that's not immediately easy to do because alternate_name is only easy to add within Client tests (src/test/client/) and not in a broader test where disaster recovery is also performed. One option is to add libcephfs/python bindings for a script but that's not trivial. Or a test-specific asok command. Any other thoughts?

vshankar · 2024-02-28T14:03:17Z

Right now the cephfs-journal-tool always uses the legacy encoding for dentries which will drop the alternate_name if ever set.

Good catch!

batrick · 2024-02-28T14:40:25Z

Obviously, I think tests need written for this but unfortunately that's not immediately easy to do because alternate_name is only easy to add within Client tests (src/test/client/) and not in a broader test where disaster recovery is also performed. One option is to add libcephfs/python bindings for a script but that's not trivial. Or a test-specific asok command. Any other thoughts?

@vshankar @lxbsz is the fs:fscrypt suite stable enough for adding a test?

gregsfortytwo · 2024-02-28T14:51:21Z

src/tools/cephfs/JournalTool.cc

-        encode('I', dentry_bl);
+        encode('i', dentry_bl);
+        ENCODE_START(2, 1, dentry_bl);
+        encode(fb.alternate_name, dentry_bl);


Is this really the encode order, where Inode has alternate_name at the front and Link has it at the back?

In both cases alternate_name is a piece of dentry metadata. As a primary link, ino/d_type are inferred from the inode that's embedded in the dentry. It doesn't really matter in any case (as inode encoding was also made versioned [1] where before it was not [2]) but made sense to everyone at the time.

[1] https://github.com/ceph/ceph/pull/37297/files#diff-c0b626494569135ea1406cfad7ab9929a4df0a1f7d880405b5d2e734db812a03R2207-R2241
[2] https://github.com/ceph/ceph/pull/37297/files#diff-c0b626494569135ea1406cfad7ab9929a4df0a1f7d880405b5d2e734db812a03L2285-L2309

This is correct.

chrisphoffman · 2024-03-13T21:13:53Z

Obviously, I think tests need written for this but unfortunately that's not immediately easy to do because alternate_name is only easy to add within Client tests (src/test/client/) and not in a broader test where disaster recovery is also performed. One option is to add libcephfs/python bindings for a script but that's not trivial. Or a test-specific asok command. Any other thoughts?

I think asok could work in this case. How much encoding validation do we currently do?

Per this here [1] , this field is currently used for supporting long file names on clients who don't know about fscrypt. I proposed a custom client to validate security posture in [2]. Perhaps we add additional tests to this custom client, to further test when that becomes available.

1: https://lwn.net/Articles/843302/
2: #54725 (comment)

batrick · 2024-03-15T21:06:32Z

Obviously, I think tests need written for this but unfortunately that's not immediately easy to do because alternate_name is only easy to add within Client tests (src/test/client/) and not in a broader test where disaster recovery is also performed. One option is to add libcephfs/python bindings for a script but that's not trivial. Or a test-specific asok command. Any other thoughts?

I think asok could work in this case. How much encoding validation do we currently do?

I'd really rather not create an asok command just for synthetic manipulation of alternate_name of a dentry, if possible.

Per this here [1] , this field is currently used for supporting long file names on clients who don't know about fscrypt. I proposed a custom client to validate security posture in [2]. Perhaps we add additional tests to this custom client, to further test when that becomes available.

1: https://lwn.net/Articles/843302/ 2: #54725 (comment)

No actually alternate_name is only used for the encrypted file name if it's too long for a regular file name. It has a very narrow use-case (atm, samba is thinking about using it for caps-insensitive file names).

lxbsz · 2024-03-18T00:05:28Z

Obviously, I think tests need written for this but unfortunately that's not immediately easy to do because alternate_name is only easy to add within Client tests (src/test/client/) and not in a broader test where disaster recovery is also performed. One option is to add libcephfs/python bindings for a script but that's not trivial. Or a test-specific asok command. Any other thoughts?

@vshankar @lxbsz is the fs:fscrypt suite stable enough for adding a test?

@batrick Sorry for late reply and I think missed this.

Yeah it is. The #54725 is the last piece of the puzzle. But won't affect this tests IMO.

batrick · 2024-05-17T16:49:41Z

Example failure with the new test:

2024-05-16T22:34:21.781 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump tree /dir 0
...
[
  {
    "accounted_rstat": {
      "rbytes": 4096,
      "rctime": "2024-05-16T22:34:15.251864+0000",
      "rfiles": 1,
      "rsnaps": 0,
      "rsubdirs": 1,
      "version": 0
    },
    "atime": "2024-05-16T22:34:14.498880+0000",
    "auth_pins": 0,
    "auth_state": {
      "replicas": {}
    },
    "authlock": {},
    "backtrace_version": 14,
    "btime": "2024-05-16T22:34:14.498880+0000",
    "change_attr": 3,
    "client_caps": [
      {
        "client_id": 5184,
        "issued": "pAsLsXsFsx",
        "last_sent": 4,
        "pending": "pAsLsXsFsx",
        "wanted": "pAsLsXsFsxcral"
      }
    ],
    "client_ranges": [],
    "ctime": "2024-05-16T22:34:15.249864+0000",
    "damage_flags": 0,
    "dir_layout": {
      "dir_hash": 2,
      "unused1": 0,
      "unused2": 0,
      "unused3": 0
    },
    "dirfrags": [
      {
        "auth_pins": 0,
        "auth_state": {
          "replicas": {}
        },
        "committed_version": "0",
        "committing_version": "0",
        "dentries": [
          {
            "alternate_name": "bUHUwH9E8uiVaf8xZ+zONcB1CToj53x5aUUnKdnNj5U37zbh28l1AaWwHhbOT3HyzqKjmSKKW1o4odQJc7nF9xrKIB8D3b4qqb2Cs6s7t2106hHhQk5/YV7DtpeNPZnorcTqxPM/hExtWHSS4P+S+Dpwj62hMyh/77sGhiW1Filvv1gQjV+sN/GozPNwHgfleadkUs1OkRkYtgWrCjbKP0MayRtiOLrVTRuYyOp/Qt3+XCIyiS87B9bUcOFjWratF+yR0kpJ0RYriix7NKVkBJ0kGWYSCY+PYjiLeMYJBMQcCxW/nwfVku+m6fgFJvb6pjEFxIk9zT5cunSImsjr",
            "auth_pins": 0,
            "auth_state": {
              "replicas": {}
            },
            "inode": 1099511628283,
            "is_auth": true,
            "is_freezing": false,
            "is_frozen": false,
            "is_new": false,
            "is_null": false,
            "is_primary": true,
            "is_remote": false,
            "lock": {},
            "nref": 2,
            "path": "dir/bUHUwH9E8uiVaf8xZ+zONcB1CToj53x5aUUnKdnNj5U37zbh28l1AaWwHhbOT3HyzqKjmSKKW1o4odQJc7nF9xrKIB8D3b4qqb2Cs6s7t2106hHhQk5,YV7DtpeNPZnorcTqxPM,hExtWHSS4P+S+Dpwj62hMyh,77sGhiW1Filvv1gQjV+sN,GozPNwHgfleadkUuZ+PMLCaKQXhuid9WvmHanxJnaabYDLj4VEz+EX2WsG",
...
# fail + journal recovery
2024-05-16T22:35:31.077 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump tree /dir 0
...
[
  {
    "accounted_rstat": {
      "rbytes": 4096,
      "rctime": "2024-05-16T22:34:15.251864+0000",
      "rfiles": 1,
      "rsnaps": 0,
      "rsubdirs": 1,
      "version": 0
    },
    "atime": "2024-05-16T22:34:14.498880+0000",
    "auth_pins": 0,
    "auth_state": {
      "replicas": {}
    },
    "authlock": {},
    "backtrace_version": 14,
    "btime": "2024-05-16T22:34:14.498880+0000",
    "change_attr": 3,
    "client_caps": [],
    "client_ranges": [],
    "ctime": "2024-05-16T22:34:15.249864+0000",
    "damage_flags": 0,
    "dir_layout": {
      "dir_hash": 2,
      "unused1": 0,
      "unused2": 0,
      "unused3": 0
    },
    "dirfrags": [
      {
        "auth_pins": 0,
        "auth_state": {
          "replicas": {}
        },
        "committed_version": "5",
        "committing_version": "5",
        "dentries": [
          {
            "alternate_name": "",
            "auth_pins": 0,
            "auth_state": {
              "replicas": {}
            },
            "inode": 1099511628283,
            "is_auth": true,
            "is_freezing": false,
            "is_frozen": false,
            "is_new": false,
            "is_null": false,
            "is_primary": true,
            "is_remote": false,
            "lock": {},
            "nref": 2,
            "path": "dir/bUHUwH9E8uiVaf8xZ+zONcB1CToj53x5aUUnKdnNj5U37zbh28l1AaWwHhbOT3HyzqKjmSKKW1o4odQJc7nF9xrKIB8D3b4qqb2Cs6s7t2106hHhQk5,YV7DtpeNPZnorcTqxPM,hExtWHSS4P+S+Dpwj62hMyh,77sGhiW1Filvv1gQjV+sN,GozPNwHgfleadkUuZ+PMLCaKQXhuid9WvmHanxJnaabYDLj4VEz+EX2WsG",

/teuthology/pdonnell-2024-05-16_22:05:38-fs:fscrypt-wip-pdonnell-testing-20240515.023933-debug-distro-default-smithi/7709888/teuthology.log

I'll polish this and cleanup the commits.

batrick · 2024-05-17T17:02:08Z

@lxbsz @vshankar PTAL

batrick · 2024-05-17T17:04:27Z

@chrisphoffman

src/tools/cephfs/JournalTool.cc

chrisphoffman · 2024-05-20T19:54:34Z

qa/tasks/cephfs/test_fscrypt.py

+
+        self.fs.fail()
+
+        self.fs.journal_tool(['event', 'recover_dentries', 'list'], 0)


I'm running this patch on my local dev machine. It gets to this line and fails during decode(). Have you run this on teuthology? Can you share a log for me to look at?

I have the example without the fix here:

#55792 (comment)

but I don't have a link on hand with this fix.

Are you running an old version of the journal-tool perhaps?

I've ran it via vstart_runner within branch: i64602.

Also ran the command manually:
./bin/cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event recover_dentries list

I'll run this through QA and see what we find.

So yes, there is a decode error: https://pulpito.ceph.com/pdonnell-2024-05-21_20:54:38-fs:fscrypt-wip-pdonnell-testing-20240521.172753-debug-distro-default-smithi/

It doesn't make much sense to me. I'll reproduce locally and gdb it.

This is fixed now @chrisphoffman . See:

https://github.com/ceph/ceph/pull/55792/files#diff-21e1004308449ef4639ea68b32b19951398157cece30ce6e7a48ab6b84c21456R908

The InodeStore needed encoding wrapping as well.

batrick · 2024-06-08T19:39:40Z

This PR is under test in https://tracker.ceph.com/issues/66402.

batrick · 2024-06-09T17:28:03Z

jenkins test make check arm64

* refs/pull/55792/head: tools/cephfs: recover alternate_name of dentries from journal qa: add test to verify recovery of alternate_name from journal tools/cephfs/JournalTool: amend comment to be consistent with code tools/cephfs/JournalTool: add some more debugging mds: remove extraneous 0x in debug output mds: dump alternate_name to formatter mds: add warning about encoding new fields

batrick · 2024-06-09T23:54:18Z

jenkins test make check

batrick · 2024-06-10T14:02:51Z

I think this PR is ready for general QA. The added test now suceeds:

https://pulpito.ceph.com/pdonnell-2024-06-10_00:07:27-fs:fscrypt-wip-pdonnell-testing-20240608.193933-debug-distro-default-smithi/

except the job still fails due to a known issue https://tracker.ceph.com/issues/65136

chrisphoffman

LGTM

nits:

ceph/src/tools/cephfs/JournalTool.cc

Line 902 in 95df0bc

// Compose: Dentry format is dnfirst, [i|l], InodeStore(bare=true)

update comment, below bare=False

ceph/src/tools/cephfs/JournalTool.cc

Line 964 in 95df0bc

// Compose: Dentry format is dnfirst, [I|L], InodeStore(bare=true)

update comment, change case to i|l and bare=false

It's never a good idea to put new fields in the middle of other encoded values unless you're willing to break all backwards-compatibility. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Looked like: recover_dentries: updating root 0x0x1 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Test without the fix: 2024-05-16T22:34:21.781 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump tree /dir 0 ... [ { "accounted_rstat": { "rbytes": 4096, "rctime": "2024-05-16T22:34:15.251864+0000", "rfiles": 1, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-05-16T22:34:14.498880+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 14, "btime": "2024-05-16T22:34:14.498880+0000", "change_attr": 3, "client_caps": [ { "client_id": 5184, "issued": "pAsLsXsFsx", "last_sent": 4, "pending": "pAsLsXsFsx", "wanted": "pAsLsXsFsxcral" } ], "client_ranges": [], "ctime": "2024-05-16T22:34:15.249864+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dentries": [ { "alternate_name": "bUHUwH9E8uiVaf8xZ+zONcB1CToj53x5aUUnKdnNj5U37zbh28l1AaWwHhbOT3HyzqKjmSKKW1o4odQJc7nF9xrKIB8D3b4qqb2Cs6s7t2106hHhQk5/YV7DtpeNPZnorcTqxPM/hExtWHSS4P+S+Dpwj62hMyh/77sGhiW1Filvv1gQjV+sN/GozPNwHgfleadkUs1OkRkYtgWrCjbKP0MayRtiOLrVTRuYyOp/Qt3+XCIyiS87B9bUcOFjWratF+yR0kpJ0RYriix7NKVkBJ0kGWYSCY+PYjiLeMYJBMQcCxW/nwfVku+m6fgFJvb6pjEFxIk9zT5cunSImsjr", "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511628283, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "dir/bUHUwH9E8uiVaf8xZ+zONcB1CToj53x5aUUnKdnNj5U37zbh28l1AaWwHhbOT3HyzqKjmSKKW1o4odQJc7nF9xrKIB8D3b4qqb2Cs6s7t2106hHhQk5,YV7DtpeNPZnorcTqxPM,hExtWHSS4P+S+Dpwj62hMyh,77sGhiW1Filvv1gQjV+sN,GozPNwHgfleadkUuZ+PMLCaKQXhuid9WvmHanxJnaabYDLj4VEz+EX2WsG", ... # fail + journal recovery 2024-05-16T22:35:31.077 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump tree /dir 0 ... [ { "accounted_rstat": { "rbytes": 4096, "rctime": "2024-05-16T22:34:15.251864+0000", "rfiles": 1, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-05-16T22:34:14.498880+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 14, "btime": "2024-05-16T22:34:14.498880+0000", "change_attr": 3, "client_caps": [], "client_ranges": [], "ctime": "2024-05-16T22:34:15.249864+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "5", "committing_version": "5", "dentries": [ { "alternate_name": "", "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511628283, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "dir/bUHUwH9E8uiVaf8xZ+zONcB1CToj53x5aUUnKdnNj5U37zbh28l1AaWwHhbOT3HyzqKjmSKKW1o4odQJc7nF9xrKIB8D3b4qqb2Cs6s7t2106hHhQk5,YV7DtpeNPZnorcTqxPM,hExtWHSS4P+S+Dpwj62hMyh,77sGhiW1Filvv1gQjV+sN,GozPNwHgfleadkUuZ+PMLCaKQXhuid9WvmHanxJnaabYDLj4VEz+EX2WsG", Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Right now the cephfs-journal-tool always uses the legacy encoding for dentries which will drop the alternate_name if ever set. Fixes: 39f3440 Fixes: https://tracker.ceph.com/issues/64602 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

batrick · 2024-06-10T16:56:24Z

LGTM

nits:

ceph/src/tools/cephfs/JournalTool.cc

Line 902 in 95df0bc

// Compose: Dentry format is dnfirst, [i|l], InodeStore(bare=true)

update comment, below bare=False

ceph/src/tools/cephfs/JournalTool.cc

Line 964 in 95df0bc

// Compose: Dentry format is dnfirst, [I|L], InodeStore(bare=true)

update comment, change case to i|l and bare=false

Thanks for pointing that out, fixed!

batrick · 2024-06-10T18:10:03Z

jenkins test api

batrick · 2024-06-11T01:06:24Z

jenkins test make check arm64

batrick · 2024-06-11T17:04:30Z

This PR is under test in https://tracker.ceph.com/issues/66433.

batrick · 2024-06-13T01:49:43Z

This PR is under test in https://tracker.ceph.com/issues/66462.

* refs/pull/55792/head: tools/cephfs: recover alternate_name of dentries from journal qa: add test to verify recovery of alternate_name from journal tools/cephfs/JournalTool: add some more debugging tools/cephfs/JournalTool: remove extraneous 0x in debug output mds: dump alternate_name to formatter mds: add warning about encoding new fields

batrick · 2024-06-19T19:18:11Z

2024-06-13T14:43:04.368 INFO:tasks.cephfs_test_runner:That alternate_name can be recovered from the journal. ... ok

from http://qa-proxy.ceph.com/teuthology/pdonnell-2024-06-13_12:32:20-fs-wip-pdonnell-testing-20240613.014923-debug-distro-default-smithi/7753792/teuthology.log

batrick added cephfs Ceph File System tools needs-review labels Feb 27, 2024

batrick requested review from chrisphoffman and lxbsz February 28, 2024 02:25

vshankar requested a review from a team February 28, 2024 13:47

gregsfortytwo reviewed Feb 28, 2024

View reviewed changes

batrick force-pushed the i64602 branch 2 times, most recently from 5c49e33 to 5aa7484 Compare May 15, 2024 02:39

github-actions bot added the tests label May 15, 2024

batrick force-pushed the i64602 branch 7 times, most recently from 69a8de6 to 4d9eb97 Compare May 16, 2024 22:05

batrick force-pushed the i64602 branch 2 times, most recently from 816c2ae to 7cc6bdc Compare May 17, 2024 17:00

batrick requested a review from vshankar May 17, 2024 17:01

chrisphoffman reviewed May 20, 2024

View reviewed changes

batrick force-pushed the i64602 branch from 92206ee to 682e1f1 Compare June 9, 2024 20:36

batrick force-pushed the i64602 branch from 682e1f1 to 95df0bc Compare June 10, 2024 13:59

batrick removed the wip-pdonnell-testing3 label Jun 10, 2024

chrisphoffman approved these changes Jun 10, 2024

View reviewed changes

batrick added 6 commits June 10, 2024 12:52

mds: add warning about encoding new fields

d27b115

It's never a good idea to put new fields in the middle of other encoded values unless you're willing to break all backwards-compatibility. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: dump alternate_name to formatter

a66e80a

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

tools/cephfs/JournalTool: remove extraneous 0x in debug output

7599d47

Looked like: recover_dentries: updating root 0x0x1 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

tools/cephfs/JournalTool: add some more debugging

4b4cb54

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

batrick force-pushed the i64602 branch from 95df0bc to bebec1f Compare June 10, 2024 16:56

batrick added needs-qa wip-pdonnell-testing labels Jun 10, 2024

batrick merged commit 959f007 into ceph:main Jun 19, 2024

batrick deleted the i64602 branch June 19, 2024 19:20

This was referenced Jun 24, 2024

reef: tools/cephfs: recover alternate_name of dentries from journal #58232

Merged

squid: tools/cephfs: recover alternate_name of dentries from journal #58233

Merged

quincy: tools/cephfs: recover alternate_name of dentries from journal #58234

Closed


		self.fs.fail()

		self.fs.journal_tool(['event', 'recover_dentries', 'list'], 0)

Conversation

batrick commented Feb 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

batrick commented Feb 27, 2024

Uh oh!

vshankar commented Feb 28, 2024

Uh oh!

batrick commented Feb 28, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrisphoffman commented Mar 13, 2024

Uh oh!

batrick commented Mar 15, 2024

Uh oh!

lxbsz commented Mar 18, 2024

Uh oh!

batrick commented May 17, 2024

Uh oh!

batrick commented May 17, 2024

Uh oh!

batrick commented May 17, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

batrick commented Jun 8, 2024

Uh oh!

batrick commented Jun 9, 2024

Uh oh!

batrick commented Jun 9, 2024

Uh oh!

batrick commented Jun 10, 2024

Uh oh!

chrisphoffman left a comment

Choose a reason for hiding this comment

Uh oh!

batrick commented Jun 10, 2024

Uh oh!

batrick commented Jun 10, 2024

Uh oh!

batrick commented Jun 11, 2024

Uh oh!

batrick commented Jun 11, 2024

Uh oh!

batrick commented Jun 13, 2024

Uh oh!

batrick commented Jun 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

batrick commented Feb 27, 2024 •

edited

Loading