Skip to content

Easy api#255

Merged
eddiebergman merged 51 commits intomainfrom
easy_api
Aug 15, 2022
Merged

Easy api#255
eddiebergman merged 51 commits intomainfrom
easy_api

Conversation

@eddiebergman
Copy link
Copy Markdown
Contributor

@eddiebergman eddiebergman commented Jun 14, 2022 •
edited
Loading

This PR aims to

  1. Add some functionality to make creating config spaces less verbose
  2. Maintain a pure python API layer for creation
  3. Unify disparities between constructions of types of parameters.
  4. Update documentation
  5. Not break any old code

To facilitate automl/SMAC3#847, in which the user needs to know about ConfigSpace before they can even try SMAC, this PR allows basic ConfigSpace's to be built using a dict

cs = ConfigurationSpace({
    "a": (0.1, 1.5),  # UniformFloat
    "b": (2, 10),  # UniformInt
    "c": ["cat", "dog", "mouse"]  #Categorical
})
  • Only the above three are simple to support and rather than complicate this and parsing logic, users could use the next API points to "progress" from the easiest use case
  • Using set vs list to distinguish between Categorical and Ordinal wasn't possible as Categorical's state that they are indeterminate w.r.t. to having a set. This might be possible to solve by simply sorting the received set in Categorical but outside of scope.

Another thing in this PR is to provide a simpler interface for constructing hyperparameters.

  1. Unifies their construction
  2. Less verbose
  3. Leaves open the distribution to be separate from the hyperparameter type
from ConfigSpace import Integer, Float, Categorical, Uniform, Normal, Beta

# Similar arguments as you're used to
x = Integer("a", (2, 20))
x = Integer("a", (2, 2000), log=True, quantization=3, default=3)

# Almost identical for Float ...
x = Float("a", (2.34, 2000), log=True, quantization=3, default=3)

a = Categorical("a", ["cat", "dog", "mouse"])
b = Categorical("b", ["small", "medium", "large"], ordered=True)  # Use the ordered flag

# Distributions
a = Integer("a", (1, 10), distribution=Uniform()) 
b = Integer("b", distribution=Normal(1, 10))  # Could also specify bounds
c = Integer("c", distribution=Beta(1, 10))

These are type safe so Mypy correctly knows that b is an OrdinalHyperparamter here, as well as for the rest of them.

The two of these combine to form:

from ConfigSpace import ConfigurationSpace

cs = ConfigurationSpace({
    # Simple stuff stays simple
    "edge": ["sigmoid", "relu", ...],
    "depth": (1, 10),
    
    # Level up, you need more
    "lr": Float("lr", (0.1, 1), log=True, default=0.2),
    "other": Category("other", ["a", "b", "c"], weights=[1, 5, 1]),
    
    # Still use old ones too
    "old_param": UniformIntegerHyperparamter("old_param", ...)
})

# Or as keyword arguments
cs = ConfigurationSpace(
    seed=1234,
    space={ ... }
)

This also updates all documentation to reflect this (as well as update documentation in general)

You can run the following to download, build and view the docs:

git clone "https://github.com/automl/ConfigSpace" && cd ConfigSpace && git checkout easy_api && python -m venv .config_space_venv_tmp && source .config_space_venv_tmp/bin/activate && pip install -e ".[dev]" && make docs && rm -rf .config_space_venv_tmp && deactivate && python -c "import webbrowser; webbrowser.open('docs/build/html/index.html')"

Note that it has to unfortunately build the package first because Cython (grrr) needs to read the docstrings and generate something? That's why it's so long but the command mostly cleans up after itself, you just have to delete the ConfigSpace folder created once you're done. Might this might not work on Windows/Mac

@eddiebergman
Copy link
Copy Markdown
Contributor Author

eddiebergman commented Jun 14, 2022

I'll clean up things once there's some feedback :)

@codecov
Copy link
Copy Markdown

codecov bot commented Jun 14, 2022 •
edited
Loading

Codecov Report

Merging #255 (bc31b65) into main (bbdb2aa) will increase coverage by 0.92%.
The diff coverage is 83.03%.

@@            Coverage Diff             @@
##             main     #255      +/-   ##
==========================================
+ Coverage   66.72%   67.64%   +0.92%     
==========================================
  Files          17       24       +7     
  Lines        1665     1768     +103     
==========================================
+ Hits         1111     1196      +85     
- Misses        554      572      +18
Impacted Files Coverage Δ
ConfigSpace/read_and_write/json.py 80.00% <ø> (ø)
ConfigSpace/read_and_write/pcs.py 85.53% <ø> (ø)
ConfigSpace/nx/algorithms/dag.py 45.91% <50.00%> (ø)
ConfigSpace/api/types/float.py 75.00% <75.00%> (ø)
ConfigSpace/api/types/integer.py 75.00% <75.00%> (ø)
ConfigSpace/api/types/categorical.py 80.00% <80.00%> (ø)
ConfigSpace/__authors__.py 100.00% <100.00%> (ø)
ConfigSpace/__init__.py 100.00% <100.00%> (ø)
ConfigSpace/__version__.py 100.00% <100.00%> (ø)
ConfigSpace/api/__init__.py 100.00% <100.00%> (ø)
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@eddiebergman
Copy link
Copy Markdown
Contributor Author

eddiebergman commented Jun 14, 2022

@nabenabe0928 @NeoChaos12 Feel free to also provide feedback on the API

@eddiebergman eddiebergman self-assigned this Jun 14, 2022
@eddiebergman eddiebergman requested a review from DaStoll June 14, 2022 14:31
@renesass
Copy link
Copy Markdown
Contributor

renesass commented Jun 17, 2022

I really like how you can interact with configspace right now. I added examples in both readme and docs.
Good work, Edds!

@eddiebergman
Copy link
Copy Markdown
Contributor Author

eddiebergman commented Jun 27, 2022

Waiting to rebase onto #259 to fix errors in tests

mfeurer
mfeurer reviewed Aug 8, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@mfeurer mfeurer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey folks, I started having a look at this PR, but I have a few questions before I can continue looking further. In any case, it looks very nice and I like the new API very much.

@renesass
Copy link
Copy Markdown
Contributor

renesass commented Aug 9, 2022

One thing which came up earlier in my mind was that we have

  • Int
  • Float
  • Categorical

Int, in contrast to float and categorical, is shortcut. Should we change it to Integer for consistency?

@mfeurer
Copy link
Copy Markdown
Contributor

mfeurer commented Aug 9, 2022

Int, in contrast to float and categorical, is shortcut. Should we change it to Integer for consistency?

Sound good.

@eddiebergman
Copy link
Copy Markdown
Contributor Author

eddiebergman commented Aug 9, 2022 •
edited
Loading

Well I don't think Float has a well known shortcut and Cat is a different word but sure, I don't really mind. I was going with programming syntax words.
I\ll change them now if you both agree on it.

@mfeurer
Copy link
Copy Markdown
Contributor

mfeurer commented Aug 9, 2022

I had an offline discussion with @eddiebergman and we agreed to add the doctests back in. I will do a full review after these changes.

@eddiebergman
Copy link
Copy Markdown
Contributor Author

eddiebergman commented Aug 10, 2022 •
edited
Loading

@mfeurer I've updated to doctests again. Took a long ass while to debug why things weren't working but make doc now correctly does everything. We have to rebuild the package for the docstrings to properly be exported and doctested which is annoying but I don't see a way around it.

I've also included the changelog in this PR since it bumps the version number, theoretically this could be done in a seperate PR but I wanted to just get it done in one place.

Seems there is also some tolerance needed for a test AssertEqual -> AssertAlmostEqual to get bumped for one test (15th decimal place). Again, could be done as a seperate PR but for expediency, its included in this.

@renesass just a note when changing from master to main which you might do elsewhere, make sure to check the docs and workflow files for references to master and change them as well.

Int was also changed to Integer everywhere

Assuming all tests checks now pass, I think it's good to go

KEggensperger
KEggensperger approved these changes Aug 12, 2022
View reviewed changes
mfeurer
mfeurer approved these changes Aug 12, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@mfeurer mfeurer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very minor, looks good to go :)

@eddiebergman
Copy link
Copy Markdown
Contributor Author

eddiebergman commented Aug 12, 2022

Extra few things done, only possible slowdown is the "docs" workflow, I hope that doesn't catch. There was a numerical thing to do with some floating point value and I hope it doesn't trigger

@eddiebergman eddiebergman merged commit 8d64a8d into main Aug 15, 2022
@eddiebergman eddiebergman deleted the easy_api branch August 15, 2022 06:43
github-actions bot pushed a commit that referenced this pull request Aug 15, 2022
Eddie Bergman: Easy api (#255)
db427c5
@eddiebergman eddiebergman mentioned this pull request Aug 19, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mfeurer mfeurer mfeurer approved these changes

@KEggensperger KEggensperger KEggensperger approved these changes

@renesass renesass Awaiting requested review from renesass

@DaStoll DaStoll Awaiting requested review from DaStoll

Assignees

@eddiebergman eddiebergman

Labels

None yet

Projects

SMAC board
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@eddiebergman @renesass @mfeurer @KEggensperger