Skip to content

Introduce a single SBOM document#606

Merged
wagoodman merged 16 commits intomainfrom
single-sbom-document
Nov 5, 2021
Merged

Introduce a single SBOM document#606
wagoodman merged 16 commits intomainfrom
single-sbom-document

Conversation

@wagoodman
Copy link
Copy Markdown
Contributor

@wagoodman wagoodman commented Oct 29, 2021

This PR replaces poweruser.JSONDocumentConfig with sbom.SBOM. Specifically this datastructure is being promoted as a first-class object to be used internally for tasking, presentation, and processing concerns. This is proposed in #555 (comment) but closely affects #554 and #556.

The main proposal of the PR is the following datastructure:

syft/syft/sbom/sbom.go

Lines 10 to 23 in a26e9e4

type SBOM struct {
Artifacts Artifacts
Source source.Metadata
}
type Artifacts struct {
PackageCatalog *pkg.Catalog
FileMetadata map[source.Location]source.FileMetadata
FileDigests map[source.Location][]file.Digest
FileClassifications map[source.Location][]file.Classification
FileContents map[source.Location]string
Secrets map[source.Location][]file.SearchResult
Distro *distro.Distro
}

All other changes attempt to promote usage of this data structure over passing parts of an SBOM around, for example... replacing:

func Encode(catalog *pkg.Catalog, metadata *source.Metadata, dist *distro.Distro, scope source.Scope, option format.Option) ([]byte, error) { ... }

with:

func Encode(s sbom.SBOM, option format.Option) ([]byte, error) { ... }

Notes:

@wagoodman wagoodman requested a review from a team October 29, 2021 16:40
@wagoodman wagoodman self-assigned this Oct 29, 2021
@github-actions
Copy link
Copy Markdown

github-actions bot commented Oct 29, 2021 •
edited
Loading

Benchmark Test Results

Benchmark results from the latest changes vs base branch 
name                                                   old time/op    new time/op    delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2          1.37ms ± 3%    1.67ms ± 2%  +21.98%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2        3.29ms ± 6%    3.88ms ± 6%  +18.15%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     821µs ± 2%     996µs ± 3%  +21.35%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 893µs ± 4%    1077µs ± 2%  +20.60%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                  848µs ± 1%    1013µs ± 2%  +19.42%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                  11.8ms ± 3%    14.2ms ± 1%  +20.49%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                 1.22ms ± 7%    1.55ms ± 3%  +26.69%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2       654ns ± 4%     783ns ± 3%  +19.65%  (p=0.008 n=5+5)

name                                                   old alloc/op   new alloc/op   delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2           248kB ± 0%     247kB ± 0%   -0.38%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2        1.11MB ± 0%    1.11MB ± 0%   -0.33%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     199kB ± 0%     197kB ± 0%   -0.68%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 228kB ± 0%     228kB ± 0%   -0.39%  (p=0.016 n=5+4)
ImagePackageCatalogers/rpmdb-cataloger-2                  222kB ± 0%     221kB ± 0%   -0.48%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                  3.24MB ± 0%    3.24MB ± 0%   -0.13%  (p=0.032 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                 1.29MB ± 0%    1.29MB ± 0%   -0.08%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2        336B ± 0%      336B ± 0%     ~     (all equal)

name                                                   old allocs/op  new allocs/op  delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2           6.82k ± 0%     6.79k ± 0%     ~     (p=0.079 n=4+5)
ImagePackageCatalogers/python-package-cataloger-2         26.3k ± 0%     26.3k ± 0%   -0.27%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     5.19k ± 0%     5.17k ± 0%   -0.46%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 6.67k ± 0%     6.65k ± 0%   -0.36%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                  6.56k ± 0%     6.53k ± 0%   -0.37%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                   59.0k ± 0%     58.9k ± 0%   -0.20%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                  7.74k ± 0%     7.71k ± 0%   -0.30%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2        9.00 ± 0%      9.00 ± 0%     ~     (all equal)

@wagoodman wagoodman mentioned this pull request Oct 31, 2021
Merged
@wagoodman wagoodman marked this pull request as ready for review November 2, 2021 14:23
@spiffcs spiffcs mentioned this pull request Nov 2, 2021
Closed
spiffcs
spiffcs reviewed Nov 2, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@spiffcs spiffcs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass.

I like the new SBOM struct and how it just encapsulates current information. Is there a future where the underlying Artifacts struct is changed as we build into developing relationships between everything?

Also small comments on data loss across the new outputs of the golden snapshots. I wasn't sure if we wanted to start excluding the information, or if it got lost as a consequence of moving to the new encode architecture.

@wagoodman wagoodman force-pushed the single-sbom-document branch 2 times, most recently from bf6a2cd to 4757c25 Compare November 2, 2021 17:39
@wagoodman
Copy link
Copy Markdown
Contributor Author

wagoodman commented Nov 2, 2021

@spiffcs

Is there a future where the underlying Artifacts struct is changed as we build into developing relationships between everything?

Indeed! I have some of that drafted out in #607, specifically:

syft/syft/sbom/sbom.go

Line 13 in c3a65f5

Relationships []artifact.Relationship

This is where the relationships object gets promoted to in the end (the sbom.SBOM struct).

@wagoodman
[wip] single sbom doc
c6acdc8 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
fix tests
a8b3280 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
fix more tests
6d14c3a 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
fix linting
39730ca 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
update cli tests
af773d2 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
remove scope in import path
a5df4b9 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
swap SPDX tag-value formatter to single sbom document
95bd1fd 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman wagoodman force-pushed the single-sbom-document branch from 4757c25 to 95bd1fd Compare November 2, 2021 18:24
@wagoodman
bust CLI cache
292bdfd 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman wagoodman force-pushed the single-sbom-document branch from 15a896f to 7c142e5 Compare November 2, 2021 21:41
@spiffcs spiffcs force-pushed the single-sbom-document branch 5 times, most recently from dabefda to 19b9013 Compare November 3, 2021 18:14
@spiffcs
update fixture to byte diff
197c27b 
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
@spiffcs spiffcs force-pushed the single-sbom-document branch from 19b9013 to 197c27b Compare November 3, 2021 18:41
@spiffcs
byte for byte
7e1d7e2 
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
@spiffcs
bust the cache
0bc33f5 
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
@spiffcs
who needs cache
fe6b1b6 
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
@spiffcs
add jar for testing
fbfc02d 
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
@spiffcs
no more bit flips
cb31cee 
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
@spiffcs
update apk with the delta for image and directory cases
7dabab8 
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
@spiffcs spiffcs requested a review from a team November 4, 2021 16:47
@spiffcs
restore cache workflow
327cac9 
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
luhring
luhring approved these changes Nov 4, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@luhring luhring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice work! 👏

I had just one question from curiosity

@wagoodman
Copy link
Copy Markdown
Contributor Author

wagoodman commented Nov 5, 2021

@spiffcs thanks a ton for solving what ended up being a caching issue I was having --I was starting to go a little crazy there 🙌

@wagoodman wagoodman merged commit bb0f35b into main Nov 5, 2021
@wagoodman wagoodman deleted the single-sbom-document branch November 5, 2021 14:05
@wagoodman wagoodman mentioned this pull request Nov 8, 2021
Merged
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
@wagoodman @spiffcs
Introduce a single SBOM document (anchore#606)
de462be 
* [wip] single sbom doc

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix more tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix linting

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update cli tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* remove scope in import path

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* swap SPDX tag-value formatter to single sbom document

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* bust CLI cache

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update fixture to byte diff

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* byte for byte

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* bust the cache

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* who needs cache

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* add jar for testing

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* no more bit flips

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* update apk with the delta for image and directory cases

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* restore cache workflow

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

Co-authored-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@spiffcs spiffcs spiffcs approved these changes

+1 more reviewer

@luhring luhring luhring approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@wagoodman wagoodman

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wagoodman @luhring @spiffcs