mon, osd: Adding read (aka workload, primary) balance score by JoshSalomon · Pull Request #47555 · ceph/ceph

JoshSalomon · 2022-08-11T10:42:05Z

Added a read (aka primary) balance score to the command 'ceph osd pool ls detail'
The score is based on the distribution of acting primaries so it reflects the performance limit on the current configuration (rather than the stable configuration). More information, including the stable configuration score can be retrieved in the json and xml output of the same command (using the -f flag).
Additional minor change - in order to test the read balance score I used the command 'ceph osd primary-temp' - this is a developers-only command that changes the acting primary. It was supposed to restore the acting primary to a PG by using OSD id of -1. This did not work since you can't pass negative number as OSD id. In order to fix that I added a command 'ceph osd rm-primary-temp [pgid]' that restores the default acting primary.

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

JoshSalomon · 2022-08-11T14:14:37Z

@rzarzynski - Can you see why my make check fails? It is related to a change I did, but 1 - it compiles for me and 2 - the fail is in under a crimson directory and I never touched files there. It seems that OSDMap.cc is moved and built also under the crimson tree and for some reason does not compile there. How can I fix such a problem (or how can I check it on my local machine). On my local machine I just build vstart. But the problem seems to be that a global variable that I use does not exist in the crimson built, and I need it...

JoshSalomon · 2022-08-13T18:32:56Z

@ljflores - Please see the API test failures, it seems this build can't load some python modules, I don't understand how this relates to my changes...

ljflores · 2022-08-16T05:14:33Z

jenkins retest this please

ljflores · 2022-08-16T05:15:47Z

@JoshSalomon there is a boost failure that affects make check and the API test right now that is happening across PRs. So it is unrelated to your changes.

ljflores · 2022-08-16T06:17:32Z

jenkins test api

ljflores

Giving an early approval since the draft looks overall very good! I have a few comments that you can review, and we should consider writing up a short bit of documentation that explains how the scores are calculated.

I'm thinking that the documentation can either be an added section here https://docs.ceph.com/en/latest/rados/operations/balancer/, or we can add an extra "workload" file in that general area.

ljflores · 2022-08-16T06:10:11Z

src/mon/OSDMonitor.cc

+	      OSDMap::wl_balance_info_t wlb_info;
+	      f->open_object_section("workload_balance");
+	      f->dump_float("score (acting)",
+	        osdmap.calc_wl_balance_score(g_ceph_context, pid, &wlb_info));


What you need to do here is either pass in cct as the context, or, at the top of the file, include this line: if you want to use the global context:

#define dout_context g_ceph_context

When I grepped for "g_ceph_context" under src, I found a ton of examples, such as in mgr/DaemonServer.cc, where it is defined.

ljflores · 2022-08-16T06:41:03Z

src/osd/OSDMap.cc

+    if (osds_crush_weight.count(osd)) {
+      total_weighted_prim_affinity += osds_crush_weight.at(osd) * get_primary_affinityf(osd);
+    }


Why is this "if" check necessary? If we're iterating over osds_crush_weight with a for loop, I don't see a reason why any OSD we need in line 5444 might not be in the osds_crush_weight map.

You also already have oweight, so you don't need osds_crush_weight.at(osd).

ljflores · 2022-08-16T06:49:59Z

src/osd/OSDMap.cc

  return candidates;
 }

+float OSDMap::calc_wl_balance_score(CephContext *cct, int64_t pool_id,


We should make it clear that this function returns the acting score, but updates all other scores as well. (Maybe in a comment above the function definition in OSDMap.h would be good)

ljflores · 2022-08-16T06:51:24Z

src/mon/OSDMonitor.cc

+	      OSDMap::wl_balance_info_t wlb_info;
+	      f->open_object_section("workload_balance");
+	      f->dump_float("score (acting)",
+	        osdmap.calc_wl_balance_score(g_ceph_context, pid, &wlb_info));


If I'm understanding correctly, does calling calc_wl_balance_score here return the acting score, but also update all the rest of the scores?

ljflores · 2022-08-16T14:19:48Z

src/mon/OSDMonitor.cc

+	    f->dump_int("pool_id", pid);
+	    f->dump_string("pool_name", osdmap.get_pool_name(pid));
+	    pdata.dump(f.get());
+	    if (pdata.get_type() == pg_pool_t::TYPE_REPLICATED) {


is_replicated()

rzarzynski · 2022-08-16T15:14:05Z

src/osd/OSDMap.cc

+    if (pdata.is_replicated())
+      snprintf (wl_score_str, sizeof(wl_score_str),
+		" workload_balance_score %.2f",
+		calc_wl_balance_score(g_ceph_context, pid));


FAILED: src/crimson/CMakeFiles/crimson-common.dir/__/osd/OSDMap.cc.o /usr/bin/ccache /usr/bin/clang++-14 -DBOOST_ASIO_DISABLE_CONCEPTS -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT -DCEPH_INSTALL_DATADIR=\"/usr/local/share/ceph\" -DCEPH_INSTALL_FULL_PKGLIBDIR=\"/usr/local/lib/ceph\" -DCMAKE_INSTALL_LIBDIR=\"lib\" -DHAVE_CONFIG_H -DSEASTAR_API_LEVEL=6 -DSEASTAR_DEBUG -DSEASTAR_DEBUG_SHARED_PTR -DSEASTAR_DEFAULT_ALLOCATOR -DSEASTAR_SCHEDULING_GROUPS_COUNT=16 -DSEASTAR_SHUFFLE_TASK_QUEUE -DSEASTAR_TYPE_ERASE_MORE -DWITH_SEASTAR=1 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -I/home/jenkins-build/build/workspace/ceph-pull-requests/build/src/include -I/home/jenkins-build/build/workspace/ceph-pull-requests/src -I/home/jenkins-build/build/workspace/ceph-pull-requests/src/seastar/include -I/home/jenkins-build/build/workspace/ceph-pull-requests/build/src/seastar/gen/include -isystem /home/jenkins-build/build/workspace/ceph-pull-requests/build/boost/include -isystem /home/jenkins-build/build/workspace/ceph-pull-requests/build/include -isystem /home/jenkins-build/build/workspace/ceph-pull-requests/src/xxHash -isystem /home/jenkins-build/build/workspace/ceph-pull-requests/src/rapidjson/include -isystem /home/jenkins-build/build/workspace/ceph-pull-requests/src/jaegertracing/opentelemetry-cpp/api/include -isystem /home/jenkins-build/build/workspace/ceph-pull-requests/src/jaegertracing/opentelemetry-cpp/exporters/jaeger/include -isystem /home/jenkins-build/build/workspace/ceph-pull-requests/src/jaegertracing/opentelemetry-cpp/ext/include -isystem /home/jenkins-build/build/workspace/ceph-pull-requests/src/jaegertracing/opentelemetry-cpp/sdk/include -g -Werror -fPIC -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wno-inconsistent-missing-override -Wno-mismatched-tags -Wno-unused-private-field -Wno-address-of-packed-member -Wno-unused-function -Wno-unused-local-typedef -Wno-varargs -Wno-gnu-designator -Wno-missing-braces -Wno-parentheses -Wno-deprecated-register -DCEPH_DEBUG_MUTEX -D_GLIBCXX_ASSERTIONS -fdiagnostics-color=auto -Wno-non-virtual-dtor -std=gnu++17 -U_FORTIFY_SOURCE -DSEASTAR_SSTRING -Wno-error=unused-result "-Wno-error=#warnings" -fstack-clash-protection -fsanitize=address -fsanitize=undefined -fno-sanitize=vptr -std=c++20 -MD -MT src/crimson/CMakeFiles/crimson-common.dir/__/osd/OSDMap.cc.o -MF src/crimson/CMakeFiles/crimson-common.dir/__/osd/OSDMap.cc.o.d -o src/crimson/CMakeFiles/crimson-common.dir/__/osd/OSDMap.cc.o -c /home/jenkins-build/build/workspace/ceph-pull-requests/src/osd/OSDMap.cc /home/jenkins-build/build/workspace/ceph-pull-requests/src/osd/OSDMap.cc:3959:25: error: use of undeclared identifier 'g_ceph_context' calc_wl_balance_score(g_ceph_context, pid)); ^ 1 error generated.

The compiler testifies g_ceph_context is undeclared here. This may happen in crimson as we have WITH_SEASTAR in multiple places, also around includes. How about including global/global_context.h?

To replicate locally you would need to install-deps.sh --crimson and also do_cmake.sh -DWITH_SEASTAR=ON.

@rzarzynski
For some reasons install-deps.sh --crimson did not do anything for me, I ran do-cmake.sh several times and installed the missing packages that it found, but now I get the following errors which I don't know how to solve
CMake Error at src/seastar/CMakeLists.txt:754 (message):
Sanitizers not found!

@JoshSalomon Radek has raised this PR to help with this: #47697

ljflores · 2022-08-17T15:47:01Z

@xiexingguo want to have a look? I know you've worked on the balancer in the past.

ljflores · 2022-08-22T23:42:59Z

@JoshSalomon I started a draft of documentation for this here: https://docs.google.com/document/d/1hVTbaaQd87HFkhhahfe4_HC7EQZ1KCz0f47cokQXk-o/edit#heading=h.yrnz08eo5nml

ljflores · 2022-08-22T23:43:20Z

jenkins test api

JoshSalomon · 2022-08-25T13:56:03Z

jenkins test make check

ljflores · 2022-08-25T15:49:50Z

jenkins test make check

ljflores · 2022-09-08T13:56:01Z

jenkins retest this please

ljflores · 2022-09-13T14:07:11Z

jenkins test make check

ljflores · 2022-09-13T14:38:24Z

@xiexingguo any thoughts on this PR?

ljflores · 2022-09-14T16:32:26Z

@JoshSalomon I'd say we can take this out of draft form since we agreed to do the documentation in #44565.

ljflores · 2022-12-13T00:43:29Z

jenkins retest this please

ljflores · 2022-12-13T00:46:26Z

I will test the new commit with this build: https://shaman.ceph.com/builds/ceph/wip-lflores-testing-2022-12-12-1839/ac7744ed2e2fd8a27b7b9e433f1822d6bee5dd87/

JoshSalomon · 2022-12-13T14:53:13Z

jenkins test api

github-actions · 2022-12-14T16:21:42Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

ljflores · 2022-12-14T17:00:17Z

@JoshSalomon this branch needs to be rebased now.

ljflores · 2022-12-14T21:06:18Z

@JoshSalomon to rebase your branch and resolve conflicts, follow these steps:

cd ceph
git checkout wip-prim-balance-score
Check what you named the ceph remote repository with git remote -v. In my case, it is named "ceph":

[lflores@fedora ceph]$ git remote -v
ceph	git@github.com:ceph/ceph.git (fetch)
ceph	git@github.com:ceph/ceph.git (push)
ljflores	git@github.com:ljflores/ceph.git (fetch)
ljflores	git@github.com:ljflores/ceph.git (push)

If you don't have the ceph remote repository added, add it with git remote add ceph git@github.com:ceph/ceph.git.
Rebase the branch with git pull <name from steps 3/4> main --rebase
You will see output like below. This means that there are conflicts that you need to resolve:

[lflores@fedora ceph]$ git pull ceph main --rebase
From github.com:ceph/ceph
 * branch                    main       -> FETCH_HEAD
Auto-merging src/mon/MonCommands.h
Auto-merging src/mon/OSDMonitor.cc
CONFLICT (content): Merge conflict in src/mon/OSDMonitor.cc
Auto-merging src/osd/OSDMap.cc
error: could not apply ad50be234f2... mon: Added command 'osd rm-primary-temp'
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply ad50be234f2... mon: Added command 'osd rm-primary-temp' (developers only) rm was not working + some minor code cleanups

Check which files have conflicts with git status. Mine shows the output below. Under "Changes to be committed", the files are green since there are no conflicts. But, the file under "Unmerged paths" is red, meaning that there is a conflict there. You can ignore all the other output:

[lflores@fedora ceph]$ git status
interactive rebase in progress; onto 76c5d1b9729
Last commands done (4 commands done):
   pick 583e741a007 osd: Calculating score for workload balancer.
   pick ad50be234f2 mon: Added command 'osd rm-primary-temp' (developers only) rm was not working + some minor code cleanups
  (see more in file .git/rebase-merge/done)
Next commands to do (7 remaining commands):
   pick 57bb9b3929a mon: minor word change in command description
   pick 0176f44eed6 osd: Changed wording from workload balancer to read balancer      Fixed compilation errors with crimson      Encorporated comments from the code review      Minor log messages improvements
  (use "git rebase --edit-todo" to view and edit)
You are currently rebasing branch 'JoshSalomon-wip-prim-balance-score' on '76c5d1b9729'.
  (fix conflicts and then run "git rebase --continue")
  (use "git rebase --skip" to skip this patch)
  (use "git rebase --abort" to check out the original branch)

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	modified:   src/mon/MonCommands.h
	modified:   src/osd/OSDMap.cc

Unmerged paths:
  (use "git restore --staged <file>..." to unstage)
  (use "git add <file>..." to mark resolution)
	both modified:   src/mon/OSDMonitor.cc

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   src/fmt (new commits)
	modified:   src/s3select (new commits)
	modified:   src/seastar (new commits)

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	blame.txt
	log-no-merges.txt
	log.txt
	out.txt
	qa/tasks/blame.txt
	src/blame.txt
	src/mds/blame.txt
	src/pybind/mgr/blame.txt
	src/rapidjson/

Open the file with conflicts (in this case, src/mon/OSDMonitor.cc, and grep for "HEAD". You should see something like this in the file. The top part shows what's currently in the main branch, and the bottom part shows what's in the commit that's causing conflicts:

<<<<<<< HEAD
  } else if (prefix == "osd primary-temp") {
=======
  } else if (prefix == "osd primary-temp" ||
             prefix == "osd rm-primary-temp") {
    string pgidstr;
    if (!cmd_getval(cmdmap, "pgid", pgidstr)) {
      ss << "unable to parse 'pgid' value '"
         << cmd_vartype_stringify(cmdmap.at("pgid")) << "'";
      err = -EINVAL;
      goto reply;
    }
>>>>>>> ad50be234f2 (mon: Added command 'osd rm-primary-temp')

Resolve the conflicts by removing the unwanted parts of code. Be sure to remove the lines that say <<<<< HEAD, ========, and >>>>>>.
Save the file.
Add the file with git add src/mon/OSDMonitor.cc.
Check that the file is staged with git status. Mine looks like the output below. The file is green under "Changes to be committed", meaning it is staged properly:

[lflores@fedora ceph]$ git status
interactive rebase in progress; onto 76c5d1b9729
Last commands done (4 commands done):
   pick 583e741a007 osd: Calculating score for workload balancer.
   pick ad50be234f2 mon: Added command 'osd rm-primary-temp' (developers only) rm was not working + some minor code cleanups
  (see more in file .git/rebase-merge/done)
Next commands to do (7 remaining commands):
   pick 57bb9b3929a mon: minor word change in command description
   pick 0176f44eed6 osd: Changed wording from workload balancer to read balancer      Fixed compilation errors with crimson      Encorporated comments from the code review      Minor log messages improvements
  (use "git rebase --edit-todo" to view and edit)
You are currently rebasing branch 'JoshSalomon-wip-prim-balance-score' on '76c5d1b9729'.
  (all conflicts fixed: run "git rebase --continue")

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	modified:   src/mon/MonCommands.h
	modified:   src/mon/OSDMonitor.cc
	modified:   src/osd/OSDMap.cc

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   src/fmt (new commits)
	modified:   src/s3select (new commits)
	modified:   src/seastar (new commits)

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	blame.txt
	log-no-merges.txt
	log.txt
	out.txt
	qa/tasks/blame.txt
	src/blame.txt
	src/mds/blame.txt
	src/pybind/mgr/blame.txt
	src/rapidjson/

Run git rebase --continue
Save the commit (in vim, it's :wq)
Force-push the rebase to your remote repository with git push -f <the name of **your** remote repository> wip-prim-balance-score. You have to force-push every time you rebase a branch, since it's like rewriting history.

You're done!

JoshSalomon · 2022-12-15T14:02:21Z

jenkins test api

JoshSalomon · 2022-12-15T15:16:29Z

jenkins test api

src/test/osd/TestOSDMap.cc

ljflores · 2023-01-25T18:58:49Z

jenkins test make check

src/test/osd/TestOSDMap.cc

Signed-off-by: Josh Salomon <jsalomon@redhat.com>

ljflores · 2023-02-10T15:58:06Z

Rados suite review: http://pulpito.front.sepia.ceph.com/?branch=wip-lflores-testing-2023-02-06-1529

Failures, unrelated:
1. https://tracker.ceph.com/issues/55347
2. https://tracker.ceph.com/issues/58475
3. https://tracker.ceph.com/issues/49287
4. https://tracker.ceph.com/issues/58560
5. https://tracker.ceph.com/issues/58684 -- new tracker; unrelated to PR in this run
6. https://tracker.ceph.com/issues/47838
7. https://tracker.ceph.com/issues/52316
8. https://tracker.ceph.com/issues/57600
9. https://tracker.ceph.com/issues/58690 -- new tracker; unrelated to PR in this run

Details:
1. SELinux Denials during cephadm/workunits/test_cephadm - Ceph - Orchestrator
2. test_dashboard_e2e.sh: Conflicting peer dependency: postcss@8.4.21 - Ceph - Mgr - Dashboard
3. podman: setting cgroup config for procHooks process caused: Unit libpod-$hash.scope not found - Ceph - Orchestrator
4. test_envlibrados_for_rocksdb.sh failed to subscrib repo - Infrastructure
5. paramiko.ssh_exception.SSHException: Mismatched MAC - Infrastructure
6. mon/test_mon_osdmap_prune.sh: first_pinned != trim_to - Ceph - RADOS
7. qa/tasks/mon_thrash.py: _do_thrash AssertionError len(s['quorum']) == len(mons) - Ceph - RADOS
8. thrash-erasure-code: AssertionError: wait_for_recovery timeout due to "active+recovery_unfound+degraded+remapped" pg - Ceph - RADOS
9. thrashosds: IndexError: Cannot choose from an empty sequence - Ceph - RADOS

ljflores · 2023-02-10T16:06:01Z

When you get a chance, please squash the commits so they make sense. And make sure that each commit has a "Signed-off-by: " line, as well as a proper title, i.e.: test/osd: write tests for read balance score.

`

for replicated pools. osd: Added workload balance score to the command ceph osd pool ls detail (different flavors for console output and json/xml output) mon: Added command 'osd rm-primary-temp' (developers only) rm was not working vstart: Added osd debug messages into mon log with -d flag For commands that execute methonds in OSD module Signed-off-by: Josh Salomon <jsalomon@redhat.com>

ljflores · 2023-02-14T21:39:24Z

Merged!! :D

github-actions bot added core mon labels Aug 11, 2022

JoshSalomon assigned ljflores Aug 11, 2022

JoshSalomon added the performance label Aug 11, 2022

ljflores self-requested a review August 11, 2022 13:42

ljflores changed the title ~~mon: Adding primary (aka worklod) balance score~~ mon: Adding primary (aka workload) balance score Aug 11, 2022

ljflores changed the title ~~mon: Adding primary (aka workload) balance score~~ mon, osd: Adding primary (aka workload) balance score Aug 16, 2022

ljflores approved these changes Aug 16, 2022

View reviewed changes

ljflores reviewed Aug 16, 2022

View reviewed changes

rzarzynski reviewed Aug 16, 2022

View reviewed changes

ljflores requested a review from xiexingguo August 17, 2022 15:46

JoshSalomon changed the title ~~mon, osd: Adding primary (aka workload) balance score~~ mon, osd: Adding read (aka workload) balance score Aug 21, 2022

JoshSalomon changed the title ~~mon, osd: Adding read (aka workload) balance score~~ mon, osd: Adding read (aka workload, primary) balance score Aug 21, 2022

github-actions bot added the mgr label Aug 25, 2022

ljflores marked this pull request as ready for review September 20, 2022 14:11

ljflores requested a review from a team as a code owner September 20, 2022 14:11

ljflores mentioned this pull request Oct 5, 2022

osd, tools, test/osd: implement read balancer #48010

Merged

14 tasks

ljflores removed the needs-qa label Oct 10, 2022

ljflores force-pushed the wip-prim-balance-score branch from c0e77eb to 55ce98e Compare October 11, 2022 16:15

ljflores added the wip-lflores-testing label Oct 13, 2022

github-actions bot added the needs-rebase label Dec 14, 2022

ljflores removed the wip-lflores-testing label Dec 14, 2022

JoshSalomon force-pushed the wip-prim-balance-score branch from 474038d to 8b62a76 Compare December 15, 2022 11:54

github-actions bot removed the needs-rebase label Dec 15, 2022

github-actions bot added the tests label Jan 16, 2023

ljflores reviewed Jan 17, 2023

View reviewed changes

src/test/osd/TestOSDMap.cc Outdated Show resolved Hide resolved

ljflores added the wip-lflores-testing label Jan 25, 2023

ljflores reviewed Jan 25, 2023

View reviewed changes

src/test/osd/TestOSDMap.cc Outdated Show resolved Hide resolved

ljflores reviewed Feb 1, 2023

View reviewed changes

src/test/osd/TestOSDMap.cc Show resolved Hide resolved

ljflores pushed a commit to ljflores/ceph that referenced this pull request Feb 1, 2023

All commits from score PR ceph#47555

3c7ebc2

Signed-off-by: Josh Salomon <jsalomon@redhat.com>

ljflores removed the wip-lflores-testing label Feb 10, 2023

JoshSalomon force-pushed the wip-prim-balance-score branch from 119ff55 to 8ed66d3 Compare February 14, 2023 14:49

ljflores merged commit 3777386 into ceph:main Feb 14, 2023

Conversation

JoshSalomon commented Aug 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

JoshSalomon commented Aug 11, 2022

Uh oh!

JoshSalomon commented Aug 13, 2022

Uh oh!

ljflores commented Aug 16, 2022

Uh oh!

ljflores commented Aug 16, 2022

Uh oh!

ljflores commented Aug 16, 2022

Uh oh!

ljflores left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ljflores commented Aug 17, 2022

Uh oh!

ljflores commented Aug 22, 2022

Uh oh!

ljflores commented Aug 22, 2022

Uh oh!

JoshSalomon commented Aug 25, 2022

Uh oh!

ljflores commented Aug 25, 2022

Uh oh!

ljflores commented Sep 8, 2022

Uh oh!

ljflores commented Sep 13, 2022

Uh oh!

ljflores commented Sep 13, 2022

Uh oh!

ljflores commented Sep 14, 2022

Uh oh!

ljflores commented Dec 13, 2022

Uh oh!

ljflores commented Dec 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JoshSalomon commented Dec 13, 2022

Uh oh!

github-actions bot commented Dec 14, 2022

Uh oh!

ljflores commented Dec 14, 2022

Uh oh!

ljflores commented Dec 14, 2022

Uh oh!

JoshSalomon commented Dec 15, 2022

Uh oh!

JoshSalomon commented Dec 15, 2022

Uh oh!

Uh oh!

ljflores commented Jan 25, 2023

Uh oh!

Uh oh!

Uh oh!

ljflores commented Feb 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JoshSalomon commented Aug 11, 2022 •

edited

Loading

ljflores commented Dec 13, 2022 •

edited

Loading

ljflores commented Feb 10, 2023 •

edited

Loading