Add relationships for python poetry packages#2906
Conversation
There was a problem hiding this comment.
I found that the internal string set is not being used, and also had a need for another set implementation for artifact.ID. I decided to simply replace this implementation with a generic one.
This comment was marked as outdated.
This comment was marked as outdated.
b05f536 to
27eee8c
Compare
5651e5b to
616ab6e
Compare
616ab6e to
9bbb4ab
Compare
9bbb4ab to
2cc1ff4
Compare
|
|
||
| // Variants allows for specifying multiple sets of provides/requires for a single package. This is useful | ||
| // in cases when you have conditional optional dependencies for a package. | ||
| Variants []Specification |
There was a problem hiding this comment.
note: a design variation of this is to instead change the dependency.Specifier to return []Specification instead of just Specification, which would be simpler, but would require more changes to the existing code base -- any thoughts/opinions on this?
There was a problem hiding this comment.
I think I have a question before I decide on which variation to land on.
Should dependency.Specifier ever be able to return more than a single instance of Specification, and if so how are those deemed to be separate instances of a Specification?
Since Specification already allows for multiple Provides []string, Requires []string
then
If dependency.Specifier returns []Specification with two distinct items:
s0 := Specification{
Provides: [foo, bar]
Requires: [baz]
}
s1 := Specification{
Provides: [faa, bas]
Requires: [bat]
}
Why are those returned as separate Specification vs the combined?
s0 := Specification{
Provides: [foo, bar, faa, bas]
Requires: [baz, bat]
}
Basically, at this point I don't think I understand yet circumstances where a Specifier would return a list vs a single merged instance when the struct doesn't have any other fields that label its origin or unique value that it originated from
There was a problem hiding this comment.
Short answer:if you combine these together into a single specification then you don't know which conditional "extra" is associated with which "provides" hint.
I think the python package extras features motivates this nicely. You can install a python package foo with pip install foo. But there may be optional features that you may choose to install (provided by the package)... say this package has the ability to work with the aws api, but not by default, since it wouldn't install the boto3 aws sdk... unless you specify the correct "extra" indication in the installation step:
pip install foo[aws]
This assumes that the package author provides the extra called "aws". This extra comes registered a set of required packages that also must be installed... such as boto3. So:
pip install foo
# install standard dependencies (baz)
pip install foo[aws]
# install standard dependencies (baz) + boto3 package and it's dependenciesSo! that means, in terms of the dependency.Specification this means that the foo package can be broken into distinct requirements:
fooStandard := Specification{
Provides: []string{"foo"},
Requires: []string{"baz"},
}
fooWithAwsPartial := Specification{
Provides: []string{"foo[aws]"},
Requires: []string{"boto3"},
}This allows for matching on each extra separately (so pip install foo[aws,gcp,azure] we'd look for 4 package specs: foo, foo[aws], foo[gcp], foo[azure]).
If a user only specifies foo[gcp], and we combined all 4 together... then it would appear that the specification both provides all variants and requires all dependencies, which is not true!
// this is not true
all := Specification{
Provides: []string{"foo", "foo[aws]", "foo[gcp]", "foo[azure]"},
Requires: []string{"batz", "boto3", "gcp-sdk", "azure-sdk"},
}What we're really looking for is:
standard := Specification{
Provides: []string{"foo"},
Requires: []string{"batz"},
}
aws := Specification{
Provides: []string{"foo[aws]"},
Requires: []string{"boto3"},
}
gcp := Specification{
Provides: []string{"foo[gcp]"},
Requires: []string{"gcp-sdk"},
}
azure := Specification{
Provides: []string{"foo[azure]"},
Requires: []string{"azure-sdk"},
}Where the assumption is that the matching process will specifically look for all parts of the request... so a user input of foo[aws] will result in a specification search for both foo and foo[aws] partials.
The current data structure allows for:
all := Specification{
Provides: []string{"foo"},
Requires: []string{"batz"},
Variant: []Specification{aws, gcp, azure}
}There was a problem hiding this comment.
After some offline discussion we decided that:
- An infinitely recursive data structure isn't useful nor simple to reason about if it happens to be leveraged (specs having specs)
- Changing
Specifierto return[]Specificationimplies that returning no specification is valid, however, technically it is not. There should always be at least a single specification that provides the package in question and requires nothing.
Instead we elected to modify the existing spec to this:
type Specification struct {
ProvidesRequires
Variants []ProvidesRequires
}
type ProvidesRequires struct {
Provides []string
Requires []string
}
spiffcs
left a comment
There was a problem hiding this comment.
🟢 given the tests look correct from my understanding of what edges we're trying to add with this PR
I asked some questions around the Specifier design decisions that you asked for feedback on.
This might be easier to talk about on a sync review where I write up the conclusion from that as the final note on this PR
|
|
||
| // Variants allows for specifying multiple sets of provides/requires for a single package. This is useful | ||
| // in cases when you have conditional optional dependencies for a package. | ||
| Variants []Specification |
There was a problem hiding this comment.
I think I have a question before I decide on which variation to land on.
Should dependency.Specifier ever be able to return more than a single instance of Specification, and if so how are those deemed to be separate instances of a Specification?
Since Specification already allows for multiple Provides []string, Requires []string
then
If dependency.Specifier returns []Specification with two distinct items:
s0 := Specification{
Provides: [foo, bar]
Requires: [baz]
}
s1 := Specification{
Provides: [faa, bas]
Requires: [bat]
}
Why are those returned as separate Specification vs the combined?
s0 := Specification{
Provides: [foo, bar, faa, bas]
Requires: [baz, bat]
}
Basically, at this point I don't think I understand yet circumstances where a Specifier would return a list vs a single merged instance when the struct doesn't have any other fields that label its origin or unique value that it originated from
I think there are still design questions in the air
|
🟢 🥳 Nice change on making the Specification structure easier to understand and not potentially infinite |
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
167f78d to
63e52fe
Compare
Partially implements #572
Creates relationships for packages found within
poetry.lockfiles. Only packages within the samepoetry.lockfile are created (no cross-file relationships allowed at this time). Honors python package "extras" so that optional additional package requirements may be specified. At this time the top-level package requirements residing in the pyproject.toml are not captured (thus if there are extras specified on dependencies there, they will not be considered). All test-fixtures added to test this behavior captures both the poetry.lock and pyproject.toml in case we want to extend this functionality in the future.In order to support python "extras", the generic
dependency.Specificationanddependency.Resolverhave been amended to consider specification variants. These variants represent all additional requirement configurations based on conditional names being present. In python's case this is represented as apackagespec and apackage[extra-name-goes-here]spec nested onto the first spec. A python package that provides multiple extras must make a variant spec for each extra, and a package that requires more than one extra must specify a requirement for each extra individually (e.g.package[extra1]andpackage[extra2]notpackages[extra1,extra2]).