FEA add (single) Cholesky Newton solver to GLMs by lorentzenchr · Pull Request #24637 · scikit-learn/scikit-learn

lorentzenchr · 2022-10-11T18:48:50Z

Reference Issues/PRs

What does this implement/fix? Explain your changes.

This PR adds one Newton solver where the Newton step is obtained by a Cholesky decomposition. This solver is a good choice for n_samples >> n_features in well-behaved setting (positive definite hessian, usually attained with at least a small L2 penalty).
This is basically the same as iterated reweighted least squares (IRLS) with inner Cholesky based solver on the normal equations.

Any other comments?

This is based on FEA add Cholesky based Newton solver to GLMs #23314.
Only PoissonRegressor, GammaRegressor and TweedieRegressor have the new solver. It would be very easy to extend it to LogisticRegression as well.
For benchmarks, see FEA add Cholesky based Newton solver to GLMs #23314 (comment) and https://github.com/lorentzenchr/notebooks/blob/master/bench_glm_newton_solvers_cholesky_lsmr.ipynb

sklearn/linear_model/tests/test_linear_loss.py

sklearn/linear_model/_glm/tests/test_glm.py

ogrisel · 2022-10-24T10:11:25Z

The pylatest_pip_openblas_pandas failure has been fixed in main.

lorentzenchr · 2022-10-24T10:22:56Z

A unrelated question: Why don't we link the codecov in the CI anymore?

lorentzenchr · 2022-10-24T10:23:40Z

The pylatest_pip_openblas_pandas failure has been fixed in main.

Great. So let's hope, CI gets 🟢 .

jjerphan · 2022-10-24T12:08:37Z

All 💚! I leave @ogrisel merge if everything looks good to him.

lorentzenchr · 2022-10-24T12:23:52Z

wait a minute

lorentzenchr

One open question to go...

lorentzenchr · 2022-10-24T12:31:37Z

sklearn/linear_model/_glm/glm.py

+                # 2.2 Deal with relative gradient differences around machine precision.
+                tiny_grad = sum_abs_grad_old * eps
+                abs_grad_improvement = np.abs(sum_abs_grad - sum_abs_grad_old)
+                check = abs_grad_improvement <= tiny_grad
+                if is_verbose:
+                    print(
+                        "      check |sum(|gradient|) - sum(|gradient_old|)| <= eps * "
+                        "sum(|gradient_old|):"
+                        f" {abs_grad_improvement} <= {tiny_grad} {check}"
+                    )
+                if check:
+                    break
+                # 2.3 This is really the last resort.
+                # Check that sum(|gradient_{i-1}|) < sum(|gradient_{i-2}|)
+                #            = has_improved_sum_abs_grad_previous
+                # If now sum(|gradient_{i}|) >= sum(|gradient_{i-1}|), this iteration
+                # made things worse and we should have stopped at i-1.


I'm undecided whether we need 2.2 and 2.3 in the line search. We could just fail and resort to lbfgs.
I added those 2 criteria when I was trying fallback strategies (e.g. simple gradient steps) for situations like indefinite hessian, negative pointwise hessian values, collinear features.

Indeed. I think we can first validate this solver by merging this PR and do this study in a follow-up PR. What do you think?

Suggested change

# 2.2 Deal with relative gradient differences around machine precision.

tiny_grad = sum_abs_grad_old * eps

abs_grad_improvement = np.abs(sum_abs_grad - sum_abs_grad_old)

check = abs_grad_improvement <= tiny_grad

if is_verbose:

print(

" check |sum(|gradient|) - sum(|gradient_old|)| <= eps * "

"sum(|gradient_old|):"

f" {abs_grad_improvement} <= {tiny_grad} {check}"

)

if check:

break

# 2.3 This is really the last resort.

# Check that sum(|gradient_{i-1}|) < sum(|gradient_{i-2}|)

# = has_improved_sum_abs_grad_previous

# If now sum(|gradient_{i}|) >= sum(|gradient_{i-1}|), this iteration

# made things worse and we should have stopped at i-1.

# TODO: study removing 2.2 and 2.3 for instead resorting to lbfgs-b.

# 2.2 Deal with relative gradient differences around machine precision.

tiny_grad = sum_abs_grad_old * eps

abs_grad_improvement = np.abs(sum_abs_grad - sum_abs_grad_old)

check = abs_grad_improvement <= tiny_grad

if is_verbose:

print(

" check |sum(|gradient|) - sum(|gradient_old|)| <= eps * "

"sum(|gradient_old|):"

f" {abs_grad_improvement} <= {tiny_grad} {check}"

)

if check:

break

# 2.3 This is really the last resort.

# Check that sum(|gradient_{i-1}|) < sum(|gradient_{i-2}|)

# = has_improved_sum_abs_grad_previous

# If now sum(|gradient_{i}|) >= sum(|gradient_{i-1}|), this iteration

# made things worse and we should have stopped at i-1.

My main problem is that I can't find the settings anymore that led me to implement those conditions. I could also remove now and we can reinsert when problems arrive. Those lines are still present in #23314.

Can you please try to remove them and re-run the 2 PR examples and the benchmarks of this comment?

#23314 (comment)

If the benchmark curve still looks the same with no new no-convergence warning, then I am fine to merge the PR without those checks and consider re-adding them if we observe the ConvergenceWarning on a real dataset.

Also I still think we could extract the line search code into a standalone function with dedicated unit tests. That would probably help trigger those conditions on arbitrary optimization problems (such as Multidimensional generalisations of the Rosenbrock function).

Can be done in a follow-up PR if what I said in the previous comment does not reveal any problem by itself.

d304ce9 remove those 2 line search criteria.

sklearn/linear_model/_glm/glm.py

jjerphan · 2022-10-24T13:45:36Z

sklearn/linear_model/_glm/glm.py

+                # 2.2 Deal with relative gradient differences around machine precision.
+                tiny_grad = sum_abs_grad_old * eps
+                abs_grad_improvement = np.abs(sum_abs_grad - sum_abs_grad_old)
+                check = abs_grad_improvement <= tiny_grad
+                if is_verbose:
+                    print(
+                        "      check |sum(|gradient|) - sum(|gradient_old|)| <= eps * "
+                        "sum(|gradient_old|):"
+                        f" {abs_grad_improvement} <= {tiny_grad} {check}"
+                    )
+                if check:
+                    break
+                # 2.3 This is really the last resort.
+                # Check that sum(|gradient_{i-1}|) < sum(|gradient_{i-2}|)
+                #            = has_improved_sum_abs_grad_previous
+                # If now sum(|gradient_{i}|) >= sum(|gradient_{i-1}|), this iteration
+                # made things worse and we should have stopped at i-1.


Indeed. I think we can first validate this solver by merging this PR and do this study in a follow-up PR. What do you think?

Suggested change

# 2.2 Deal with relative gradient differences around machine precision.

tiny_grad = sum_abs_grad_old * eps

abs_grad_improvement = np.abs(sum_abs_grad - sum_abs_grad_old)

check = abs_grad_improvement <= tiny_grad

if is_verbose:

print(

" check |sum(|gradient|) - sum(|gradient_old|)| <= eps * "

"sum(|gradient_old|):"

f" {abs_grad_improvement} <= {tiny_grad} {check}"

)

if check:

break

# 2.3 This is really the last resort.

# Check that sum(|gradient_{i-1}|) < sum(|gradient_{i-2}|)

# = has_improved_sum_abs_grad_previous

# If now sum(|gradient_{i}|) >= sum(|gradient_{i-1}|), this iteration

# made things worse and we should have stopped at i-1.

# TODO: study removing 2.2 and 2.3 for instead resorting to lbfgs-b.

# 2.2 Deal with relative gradient differences around machine precision.

tiny_grad = sum_abs_grad_old * eps

abs_grad_improvement = np.abs(sum_abs_grad - sum_abs_grad_old)

check = abs_grad_improvement <= tiny_grad

if is_verbose:

print(

" check |sum(|gradient|) - sum(|gradient_old|)| <= eps * "

"sum(|gradient_old|):"

f" {abs_grad_improvement} <= {tiny_grad} {check}"

)

if check:

break

# 2.3 This is really the last resort.

# Check that sum(|gradient_{i-1}|) < sum(|gradient_{i-2}|)

# = has_improved_sum_abs_grad_previous

# If now sum(|gradient_{i}|) >= sum(|gradient_{i-1}|), this iteration

# made things worse and we should have stopped at i-1.

lorentzenchr · 2022-10-24T17:09:18Z

Check of final changes:

SKLEARN_TESTS_GLOBAL_RANDOM_SEED="all" pytest -n auto -x sklearn/linear_model/_glm/tests/test_glm.py passes locally
Benchmark plots as in FEA add Cholesky based Newton solver to GLMs #23314 (comment)
- "lbfgs" is LogisticRegression(solver="lbfgs")
- "lbfgs2 is using
```
class BinomialRegressor(_GeneralizedLinearRegressor):
def _get_loss(self):
    return HalfBinomialLoss()
```
  with solver="lbfgs"

lorentzenchr · 2022-10-24T17:14:51Z

The plot becomes even more convincing with train_size=50_000 instead of just 5000:

ogrisel · 2022-10-24T17:20:07Z

Let's wait for the CI to complete, if there is no new ConvergenceWarning in the examples, please feel free to merge!

jjerphan · 2022-10-24T20:16:38Z

No ConvergenceWarning, hence…

jjerphan · 2022-10-24T20:18:06Z

Merged!

Thank you a lot for bringing this solver in, @lorentzenchr! Thanks @ogrisel for your comprehensive review and analysis.

lorentzenchr · 2022-10-25T11:56:49Z

@jjerphan @ogrisel That was a great endeavor together with you two. Thanks for having brought this over the finish line before the 1.2 release!

I'm wondering if we should enable this solver in LogisticRegression, too, at least for binary classification problems.

ogrisel · 2022-10-25T12:00:29Z

+1 for binary logistic regression. Multinomial LR will probably require significantly more work and will be problematic when both n_classes and n_features are non small at the same time.

* FEA add NewtonSolver, CholeskyNewtonSolver and QRCholeskyNewtonSolver * ENH better singular hessian special solve * CLN fix some typos found by reviewer * TST assert ConvergenceWarning is raised * MNT add BaseCholeskyNewtonSolver * WIP colinear design in GLMs * FIX _solve_singular * FIX false unpacking in * TST add tests for unpenalized GLMs * TST fix solutions of glm_dataset * ENH add SVDFallbackSolver * CLN remove SVDFallbackSolver * ENH use gradient step for singular hessians * ENH print iteration number in warnings * TST improve test_linalg_warning_with_newton_solver * CLN LinAlgWarning fron scipy.linalg * ENH more robust hessian * ENH increase maxls for lbfgs to make it more robust * ENH add hessian_warning for too many negative hessian values * CLN some warning messages * ENH add lbfgs_step * ENH use lbfgs_step for hessian_warning * TST make them pass * TST tweek rtol for lbfgs * TST add rigoros test for GLMs * TST improve test_warm_start * ENH improve lbfgs options for better convergence * CLN fix test_warm_start * TST fix assert singular values in datasets * CLN address most review comments * ENH enable more vebosity levels for lbfgs * DOC add whatsnew * CLN remove xfail and clean a bit * CLN docstring about minimum norm * More informative repr for the glm_dataset fixture cases * Forgot to run black * CLN remove unnecessary filterwarnings * CLN address review comments * Trigger [all random seeds] on the following tests: test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * CLN add comment for lbfgs ftol=64 * machine precision * CLN XXX code comment * Trigger [all random seeds] on the following tests: test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * CLN link issue and remove code snippet in comment * Trigger [all random seeds] on the following tests: test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * CLN add catch_warnings * Trigger [all random seeds] on the following tests: test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * Trigger [all random seeds] on the following tests: test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * [all random seeds] test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * Trigger with -Werror [all random seeds] test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * ENH increase maxls to 50 * [all random seeds] test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * Revert "Trigger with -Werror [all random seeds]" This reverts commit 99f4cf9. * TST add catch_warnings to filterwarnings * TST adapt tests for newton solvers * CLN cleaner gradient step with gradient_times_newton * DOC add whatsnew * ENH always use lbfgs as fallback * TST adapt rtol * TST fix test_linalg_warning_with_newton_solver * CLN address some review comments * Improve tests related to convergence warning on collinear data * overfit -> fit * Typo in comment * Apply suggestions from code review * ENH fallback_lbfgs_solve - Do not use lbfgs steps, fall back complete to lbfgs * ENH adapt rtol * Improve test_linalg_warning_with_newton_solver * Better comments * Fixed Hessian casing and improved warning messages * [all random seeds] test_linalg_warning_with_newton_solver * Ignore ConvergenceWarnings for now if convergence is good * CLN remove counting of warnings * ENH fall back to lbfgs if line search did not converge * DOC better comment on performance bottleneck * Update GLM related examples to use the new solver * CLN address reviewer comments * EXA improve some wordings * CLN do not pop "solver in parameter constraints * CLN fix typos * DOC fix docstring * CLN remove solver newton-qr-cholesky * DOC update PR number in whatsnew * CLN address review comments * CLN remove unnecessary catch_warnings * CLN address some review comments * DOC more precise whatsnew * CLN use init_zero_coef * CLN use and test init_zero_coef * CLN address some review comments * CLN mark NewtonSolver as private by leading underscore * CLN exact comments for inner_solve * TST add test_newton_solver_verbosity * TST extend test_newton_solver_verbosity * TST logic in test_glm_regression_unpenalized * TST use count_nonzero * CLN remove super rare line search checks * MNT move Newton solver to new file _newton_solver.py Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org> Co-authored-by: Julien Jerphanion <git@jjerphan.xyz>

sklearn/linear_model/_glm/glm.py

* FEA add NewtonSolver, CholeskyNewtonSolver and QRCholeskyNewtonSolver * ENH better singular hessian special solve * CLN fix some typos found by reviewer * TST assert ConvergenceWarning is raised * MNT add BaseCholeskyNewtonSolver * WIP colinear design in GLMs * FIX _solve_singular * FIX false unpacking in * TST add tests for unpenalized GLMs * TST fix solutions of glm_dataset * ENH add SVDFallbackSolver * CLN remove SVDFallbackSolver * ENH use gradient step for singular hessians * ENH print iteration number in warnings * TST improve test_linalg_warning_with_newton_solver * CLN LinAlgWarning fron scipy.linalg * ENH more robust hessian * ENH increase maxls for lbfgs to make it more robust * ENH add hessian_warning for too many negative hessian values * CLN some warning messages * ENH add lbfgs_step * ENH use lbfgs_step for hessian_warning * TST make them pass * TST tweek rtol for lbfgs * TST add rigoros test for GLMs * TST improve test_warm_start * ENH improve lbfgs options for better convergence * CLN fix test_warm_start * TST fix assert singular values in datasets * CLN address most review comments * ENH enable more vebosity levels for lbfgs * DOC add whatsnew * CLN remove xfail and clean a bit * CLN docstring about minimum norm * More informative repr for the glm_dataset fixture cases * Forgot to run black * CLN remove unnecessary filterwarnings * CLN address review comments * Trigger [all random seeds] on the following tests: test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * CLN add comment for lbfgs ftol=64 * machine precision * CLN XXX code comment * Trigger [all random seeds] on the following tests: test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * CLN link issue and remove code snippet in comment * Trigger [all random seeds] on the following tests: test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * CLN add catch_warnings * Trigger [all random seeds] on the following tests: test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * Trigger [all random seeds] on the following tests: test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * [all random seeds] test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * Trigger with -Werror [all random seeds] test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * ENH increase maxls to 50 * [all random seeds] test_glm_regression test_glm_regression_hstacked_X test_glm_regression_vstacked_X test_glm_regression_unpenalized test_glm_regression_unpenalized_hstacked_X test_glm_regression_unpenalized_vstacked_X test_warm_start * Revert "Trigger with -Werror [all random seeds]" This reverts commit 99f4cf9. * TST add catch_warnings to filterwarnings * TST adapt tests for newton solvers * CLN cleaner gradient step with gradient_times_newton * DOC add whatsnew * ENH always use lbfgs as fallback * TST adapt rtol * TST fix test_linalg_warning_with_newton_solver * CLN address some review comments * Improve tests related to convergence warning on collinear data * overfit -> fit * Typo in comment * Apply suggestions from code review * ENH fallback_lbfgs_solve - Do not use lbfgs steps, fall back complete to lbfgs * ENH adapt rtol * Improve test_linalg_warning_with_newton_solver * Better comments * Fixed Hessian casing and improved warning messages * [all random seeds] test_linalg_warning_with_newton_solver * Ignore ConvergenceWarnings for now if convergence is good * CLN remove counting of warnings * ENH fall back to lbfgs if line search did not converge * DOC better comment on performance bottleneck * Update GLM related examples to use the new solver * CLN address reviewer comments * EXA improve some wordings * CLN do not pop "solver in parameter constraints * CLN fix typos * DOC fix docstring * CLN remove solver newton-qr-cholesky * DOC update PR number in whatsnew * CLN address review comments * CLN remove unnecessary catch_warnings * CLN address some review comments * DOC more precise whatsnew * CLN use init_zero_coef * CLN use and test init_zero_coef * CLN address some review comments * CLN mark NewtonSolver as private by leading underscore * CLN exact comments for inner_solve * TST add test_newton_solver_verbosity * TST extend test_newton_solver_verbosity * TST logic in test_glm_regression_unpenalized * TST use count_nonzero * CLN remove super rare line search checks * MNT move Newton solver to new file _newton_solver.py Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org> Co-authored-by: Julien Jerphanion <git@jjerphan.xyz>

lorentzenchr and others added 30 commits May 9, 2022 18:12

FEA add NewtonSolver, CholeskyNewtonSolver and QRCholeskyNewtonSolver

d4a8b4e

Merge branch 'main' into glm_newton_cholesky

ad65d89

ENH better singular hessian special solve

267d570

CLN fix some typos found by reviewer

dd5a820

TST assert ConvergenceWarning is raised

bf1828d

MNT add BaseCholeskyNewtonSolver

9783e6b

WIP colinear design in GLMs

d373e63

Merge branch 'glm_newton_cholesky' into glm-newton-colinear

bcc176e

FIX _solve_singular

c6efcef

FIX solver for singular cases in GLMs (#5)

2381298

FIX false unpacking in

d2063f7

Merge branch 'main' into glm_newton_cholesky

8d1137f

TST add tests for unpenalized GLMs

e6684c6

TST fix solutions of glm_dataset

3fb3695

ENH add SVDFallbackSolver

2b6485e

CLN remove SVDFallbackSolver

59989f3

ENH use gradient step for singular hessians

d463817

ENH print iteration number in warnings

8a108bb

TST improve test_linalg_warning_with_newton_solver

82287af

CLN LinAlgWarning fron scipy.linalg

9868a13

ENH more robust hessian

e3a2627

ENH increase maxls for lbfgs to make it more robust

0276cd9

ENH add hessian_warning for too many negative hessian values

b429472

CLN some warning messages

a85f251

ENH add lbfgs_step

c9b1200

ENH use lbfgs_step for hessian_warning

2f0ea15

TST make them pass

9ce6cf2

TST tweek rtol for lbfgs

221f611

TST add rigoros test for GLMs

aa81fb5

TST improve test_warm_start

cd06ba7

TST extend test_newton_solver_verbosity

00f7465

jjerphan reviewed Oct 24, 2022

View reviewed changes

sklearn/linear_model/tests/test_linear_loss.py Outdated Show resolved Hide resolved

sklearn/linear_model/_glm/tests/test_glm.py Show resolved Hide resolved

sklearn/linear_model/_glm/tests/test_glm.py Show resolved Hide resolved

TST logic in test_glm_regression_unpenalized

308fd88

lorentzenchr added 2 commits October 24, 2022 12:19

TST use count_nonzero

ebf930b

Merge branch 'main' into glm_newton_cholesky_only

2ffa621

lorentzenchr commented Oct 24, 2022

View reviewed changes

jjerphan reviewed Oct 24, 2022

View reviewed changes

lorentzenchr added 2 commits October 24, 2022 18:35

CLN remove super rare line search checks

d304ce9

MNT move Newton solver to new file _newton_solver.py

f002eb7

jjerphan merged commit ff9344f into scikit-learn:main Oct 24, 2022

lorentzenchr mentioned this pull request Oct 24, 2022

FEA add Cholesky based Newton solver to GLMs #23314

Closed

lorentzenchr deleted the glm_newton_cholesky_only branch October 24, 2022 21:39

lorentzenchr mentioned this pull request Oct 27, 2022

ENH add newton-cholesky solver to LogisticRegression #24767

Merged

glemaitre reviewed Nov 2, 2022

View reviewed changes

sklearn/linear_model/_glm/glm.py Show resolved Hide resolved

agramfort mentioned this pull request Nov 27, 2022

add cholesky solver from sklearn benchopt/benchmark_logreg_l2#40

Open

lorentzenchr mentioned this pull request Nov 7, 2023

ENH deal with tiny loss improvements in line search Quantco/glum#724

Merged

1 task

Uh oh!

Conversation

lorentzenchr commented Oct 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ogrisel commented Oct 24, 2022

Uh oh!

lorentzenchr commented Oct 24, 2022

Uh oh!

lorentzenchr commented Oct 24, 2022

Uh oh!

jjerphan commented Oct 24, 2022

Uh oh!

lorentzenchr commented Oct 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lorentzenchr left a comment

Choose a reason for hiding this comment

Uh oh!

lorentzenchr Oct 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jjerphan Oct 24, 2022

Choose a reason for hiding this comment

Uh oh!

lorentzenchr Oct 24, 2022

Choose a reason for hiding this comment

Uh oh!

ogrisel Oct 24, 2022

Choose a reason for hiding this comment

Uh oh!

ogrisel Oct 24, 2022

Choose a reason for hiding this comment

Uh oh!

lorentzenchr Oct 24, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jjerphan Oct 24, 2022

Choose a reason for hiding this comment

Uh oh!

lorentzenchr commented Oct 24, 2022

Uh oh!

lorentzenchr commented Oct 24, 2022

Uh oh!

ogrisel commented Oct 24, 2022

Uh oh!

jjerphan commented Oct 24, 2022

Uh oh!

jjerphan commented Oct 24, 2022

Uh oh!

lorentzenchr commented Oct 25, 2022

Uh oh!

ogrisel commented Oct 25, 2022

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lorentzenchr commented Oct 11, 2022 •

edited

Loading

lorentzenchr commented Oct 24, 2022 •

edited

Loading

lorentzenchr Oct 24, 2022 •

edited

Loading