Skip to content

MAINT DOC HGBT leave updated if loss is not smooth#26254

Merged
OmarManzoor merged 2 commits intoscikit-learn:mainfrom
lorentzenchr:hgbt_doc_smooth_loss
Aug 8, 2023
Merged

MAINT DOC HGBT leave updated if loss is not smooth#26254
OmarManzoor merged 2 commits intoscikit-learn:mainfrom
lorentzenchr:hgbt_doc_smooth_loss

Conversation

@lorentzenchr
Copy link
Copy Markdown
Member

@lorentzenchr lorentzenchr commented Apr 21, 2023 •
edited
Loading

Reference Issues/PRs

Popped up while working on #25964.

What does this implement/fix? Explain your changes.

HGBT leave updates now rely on loss.differentiable and the reasons and differences to the standard gradient boosting algo are explained.

Any other comments?

It is hard to find a reference for gradient boosting with 2nd order loss approximation (using hessians) and non-smooth losses.
Edit: https://arxiv.org/abs/1808.03064 explicitly considers the different boosting schemes and mentions the problem of non-smooth loss functions with Newton boosting.

@lorentzenchr lorentzenchr added the Quick Review For PRs that are quick to review label Apr 21, 2023
thomasjpfan
thomasjpfan reviewed Apr 21, 2023
View reviewed changes
@glevv
Copy link
Copy Markdown
Contributor

glevv commented Apr 21, 2023 •
edited
Loading

It is hard to find a reference for gradient boosting with 2nd order loss approximation (using hessians) and non-smooth losses.

Don't know if that's what you're after, but maybe xgboost and hinge
https://github.com/dmlc/xgboost/blob/master/src/objective/hinge.cu
or lightgbm and l1
https://github.com/microsoft/LightGBM/blob/ce0813efea9a12e543ce8feca4952407e1d05e86/src/objective/cuda/cuda_regression_objective.cu#L136
They obviously don't use "real" hessians.

I think catboost also has a lot of non smooth loss functions, but they don't use 2nd order optimization algorithms for them.

thomasjpfan
thomasjpfan reviewed Apr 28, 2023
View reviewed changes
thomasjpfan
thomasjpfan approved these changes Apr 29, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@thomasjpfan thomasjpfan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lorentzenchr
Copy link
Copy Markdown
Member Author

lorentzenchr commented May 2, 2023

@glevv Do you want to give this PR a review?

@glevv
Copy link
Copy Markdown
Contributor

glevv commented May 4, 2023 •
edited
Loading

@lorentzenchr Sorry, I don't think I will be able to

@lorentzenchr
Copy link
Copy Markdown
Member Author

lorentzenchr commented May 4, 2023

@glevv No problem. I thought I just ask as you seemed interested in the PR:smirk:

@lorentzenchr lorentzenchr added this to the 1.3 milestone Jun 1, 2023
@jeremiedbb jeremiedbb modified the milestones: 1.3, 1.4 Jul 6, 2023
OmarManzoor
OmarManzoor approved these changes Aug 8, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@OmarManzoor OmarManzoor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @lorentzenchr

@OmarManzoor OmarManzoor merged commit ed01199 into scikit-learn:main Aug 8, 2023
@lorentzenchr lorentzenchr deleted the hgbt_doc_smooth_loss branch August 8, 2023 17:00
TamaraAtanasoska pushed a commit to TamaraAtanasoska/scikit-learn that referenced this pull request Aug 21, 2023
@lorentzenchr @TamaraAtanasoska
MAINT DOC HGBT leave updated if loss is not smooth (scikit-learn#26254)
3ebc8e7
REDVM pushed a commit to REDVM/scikit-learn that referenced this pull request Nov 16, 2023
@lorentzenchr @REDVM
MAINT DOC HGBT leave updated if loss is not smooth (scikit-learn#26254)
5f63946
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@thomasjpfan thomasjpfan thomasjpfan approved these changes

@OmarManzoor OmarManzoor OmarManzoor approved these changes

Assignees

No one assigned

Labels

Documentation module:ensemble Quick Review For PRs that are quick to review

Projects

None yet

Milestone

1.4

Development

Successfully merging this pull request may close these issues.

5 participants

@lorentzenchr @glevv @thomasjpfan @OmarManzoor @jeremiedbb