Skip to content

Issue933 - added the ability to customize the maximum iterations for the demeaning algorithm#944

Merged
s3alfisc merged 13 commits intopy-econometrics:masterfrom
damandhaliwal:issue933
Jun 23, 2025
Merged

Issue933 - added the ability to customize the maximum iterations for the demeaning algorithm#944
s3alfisc merged 13 commits intopy-econometrics:masterfrom
damandhaliwal:issue933

Conversation

@damandhaliwal
Copy link
Copy Markdown
Member

Clean PR without .idea and any multiple commits

@damandhaliwal
Copy link
Copy Markdown
Member Author

Fixes #933. @s3alfisc updated to new PR without .idea/ or the entire history needing to merge again.

@codecov
Copy link
Copy Markdown

codecov bot commented Jun 17, 2025

Codecov Report

Attention: Patch coverage is 66.66667% with 5 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pyfixest/estimation/estimation.py 60.00% 2 Missing ⚠️
pyfixest/estimation/demean_.py 50.00% 1 Missing ⚠️
pyfixest/estimation/demean_jax_.py 0.00% 1 Missing ⚠️
pyfixest/estimation/feglm_.py 50.00% 1 Missing ⚠️
Flag Coverage Δ
core-tests 78.42% <66.66%> (+0.02%) ⬆️
tests-extended ?
tests-vs-r 16.46% <0.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
pyfixest/estimation/FixestMulti_.py 80.90% <100.00%> (+0.19%) ⬆️
pyfixest/estimation/fegaussian_.py 89.18% <ø> (ø)
pyfixest/estimation/feiv_.py 87.50% <ø> (ø)
pyfixest/estimation/felogit_.py 88.23% <ø> (ø)
pyfixest/estimation/feols_.py 84.57% <100.00%> (-6.90%) ⬇️
pyfixest/estimation/feols_compressed_.py 82.75% <ø> (ø)
pyfixest/estimation/fepois_.py 88.78% <100.00%> (+0.05%) ⬆️
pyfixest/estimation/feprobit_.py 90.24% <ø> (ø)
pyfixest/estimation/jax/demean_jax_.py 100.00% <100.00%> (ø)
pyfixest/estimation/quantreg/quantreg_.py 73.52% <ø> (ø)
... and 4 more

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@s3alfisc
Copy link
Copy Markdown
Member

Hm for some reason the JAX demeaner does not seem to fail with fixef_maxiter = 1:

    def test_demean_model_maxiter_convergence_failure(demean_func):
         """Test that demean_model fails when maxiter is too small."""
         N = 100
         rng = np.random.default_rng(42)
    
         Y = pd.DataFrame({"y": rng.normal(0, 1, N)})
         X = pd.DataFrame({"x1": rng.normal(0, 1, N)})
         # Many fixed effects to make convergence difficult
         fe = pd.DataFrame({"fe1": np.arange(N)})  # Each obs is its own FE
         weights = np.ones(N)
         lookup_dict = {}
    
         # Should fail with very small maxiter
>        with pytest.raises(ValueError, match="Demeaning failed after 1 iterations"):
E        Failed: DID NOT RAISE <class 'ValueError'>

@s3alfisc
Copy link
Copy Markdown
Member

@all-contributors please add @damandhaliwal for code

@allcontributors
Copy link
Copy Markdown
Contributor

@s3alfisc

I've put up a pull request to add @damandhaliwal! 🎉

@s3alfisc
Copy link
Copy Markdown
Member

Made a small update to the tests - with only one fixed effect, there is not much to iterate, we simply demean x and y, it's one-shot. I think with two fixed effects, this should be different.

@s3alfisc
Copy link
Copy Markdown
Member

pre-commit.ci autofix

@s3alfisc
Copy link
Copy Markdown
Member

s3alfisc commented Jun 17, 2025

Tried quite a bit but will not manage for tonight. Seems like a more subtle issue that you have uncovered ... Sorry about that, I really thought this would be more of a "quick win".

@damandhaliwal
Copy link
Copy Markdown
Member Author

I think all the tests ran before I pushed yesterday. I'll also take another look tonight and see if I can figure it out. No worries on the "easy" win, would rather have it be tough and get me to figure stuff out

@s3alfisc
Copy link
Copy Markdown
Member

s3alfisc commented Jun 17, 2025

I think all the tests ran before I pushed yesterday. I'll also take another look tonight and see if I can figure it out. No worries on the "easy" win, would rather have it be tough and get me to figure stuff out

Ah really? Maybe you did not run tests for the JAX and Rust backends? 🤔

Btw, I just had made my mind I though "let's merge with the error as it's unrelated and fix it in a separate PR" but now you've got me thinking again. Tbh I would be surprised if everything ran without error in the other branch. Unfortunately gh already deleted all the CI logs.

No worries on the "easy" win

I love easy wins 😄

@damandhaliwal
Copy link
Copy Markdown
Member Author

Btw, I just had made my mind I though "let's merge with the error as it's unrelated and fix it in a separate PR" but now you've got me thinking again. Tbh I would be surprised if everything ran without error in the other branch.

Okay, I was mistaken. I ran it again and the test does fail for demean_jax. I can look into it tomorrow, probably need to run some examples to find where it goes a different way. My guess is that it doesn't converge either, but just does not raise the same error due to some difference

@damandhaliwal
Copy link
Copy Markdown
Member Author

@s3alfisc I fixed it. Wasn't an underlying big issue in estimation. The return variable converged in demean_jax_.py was of the format jax.array but the test was expecting just a boolean so changed the return to bool(converged). Let me know what you think. Tests passed locally.

@s3alfisc
Copy link
Copy Markdown
Member

Ah awesome! Thank you @damandhaliwal. I'll do a final review then and merge.

@s3alfisc
Copy link
Copy Markdown
Member

pre-commit.ci autofix

Copy link
Copy Markdown
Member

@s3alfisc s3alfisc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, will merge now. Thank you @damandhaliwal and congrats to your first successful PR!

@s3alfisc
Copy link
Copy Markdown
Member

The linter still fails but this is unrelated to your PR, I will fix this in a separate PR =)

@s3alfisc s3alfisc merged commit f8b7866 into py-econometrics:master Jun 23, 2025
6 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants