Skip to content

monitor: collect metadata alongside the result of a stage#2215

Merged
croissanne merged 1 commit intoosbuild:mainfrom
croissanne:metadata
Oct 8, 2025
Merged

monitor: collect metadata alongside the result of a stage#2215
croissanne merged 1 commit intoosbuild:mainfrom
croissanne:metadata

Conversation

@croissanne
Copy link
Member

@croissanne croissanne commented Oct 7, 2025

Some stages generate metadata which contains information that can only be generated when a stage is running. The v2 result contains this metadata which was lost when using the JSONSeqMonitor. By passing stage metadata along to the monitor's result function, the JSONSeqMonitor can print it alongside the result.

@croissanne croissanne force-pushed the metadata branch 3 times, most recently from b2d9f17 to 831778a Compare October 7, 2025 16:03
@supakeen supakeen self-requested a review October 7, 2025 18:11
@croissanne croissanne marked this pull request as ready for review October 8, 2025 10:35
@croissanne croissanne requested a review from a team as a code owner October 8, 2025 10:35
@croissanne croissanne requested review from lzap and thozza and removed request for a team October 8, 2025 10:35
@croissanne
monitor: collect metadata alongside the result of a stage
af40f31 
Some stages generate metadata which contains information that can only
be generated when a stage is running. The v2 result contains this
metadata which was lost when using the JSONSeqMonitor. By passing stage
metadata along to the monitor's result function, the JSONSeqMonitor can
print it alongside the result.
@croissanne croissanne changed the title monitor: add metadata to stage result monitor: collect metadata alongside the result of a stage Oct 8, 2025
@supakeen
Copy link
Member

supakeen commented Oct 8, 2025

This is specifically the information that's stored when a stage calls api.metadata(...) if I understand correctly?

@croissanne
Copy link
Member Author

croissanne commented Oct 8, 2025 •
edited
Loading

This is specifically the information that's stored when a stage calls api.metadata(...) if I understand correctly?

indeed! It's currently exposed with the LogMonitor via the result dump at the end

osbuild/osbuild/formats/v2.py

Lines 418 to 423 in 0a0a19c

for stage in p.stages:
md = obj.meta.get(stage.id)
if not md:
continue
val = data.setdefault(stage.name, {})
val.update(md)

but it's not part of the JSONSeqMonitor.

mvo5
mvo5 approved these changes Oct 8, 2025
View reviewed changes
Copy link
Contributor

@mvo5 mvo5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Looks great!

supakeen
supakeen approved these changes Oct 8, 2025
View reviewed changes
Copy link
Member

@supakeen supakeen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you looks good.

@croissanne croissanne enabled auto-merge (rebase) October 8, 2025 13:31
@croissanne croissanne merged commit 8212a53 into osbuild:main Oct 8, 2025
54 checks passed
croissanne added a commit to croissanne/images that referenced this pull request Oct 28, 2025
@croissanne
osbuild/monitor: increase status scanner buffer size to 16MiB
ec76375 
The size of the status message from the org.osbuild.rpm stage in the os
pipeline depends on how many packages a user has requested on top of the
base set. As a result, the monitor will always be breakable, but let's
increase the buffer size substantially to make it very unlikely that
users will hit this.

Especially with the addition of metadata [1], the status entry size grew
substantially for rpm stages. The rpm stage has an entry in the metadata
for each package that was installed (in addition to a very large output
message).

[1]: osbuild/osbuild#2215
croissanne added a commit to croissanne/images that referenced this pull request Oct 28, 2025
@croissanne
osbuild/monitor: increase status scanner buffer size to 16MiB
e12fc57 
The size of the status message from the org.osbuild.rpm stage in the os
pipeline depends on how many packages a user has requested on top of the
base set. As a result, the monitor will always be breakable, but let's
increase the buffer size substantially to make it very unlikely that
users will hit this.

Especially with the addition of metadata [1], the status entry size grew
substantially for rpm stages. The rpm stage has an entry in the metadata
for each package that was installed (in addition to a very large output
message).

[1]: osbuild/osbuild#2215
croissanne added a commit to croissanne/images that referenced this pull request Oct 28, 2025
@croissanne
osbuild/monitor: increase status scanner buffer size to 16MiB
1062b32 
The size of the status message from the org.osbuild.rpm stage in the os
pipeline depends on how many packages a user has requested on top of the
base set. As a result, the monitor will always be breakable, but let's
increase the buffer size substantially to make it very unlikely that
users will hit this.

Especially with the addition of metadata [1], the status entry size grew
substantially for rpm stages. The rpm stage has an entry in the metadata
for each package that was installed (in addition to a very large output
message).

[1]: osbuild/osbuild#2215
croissanne added a commit to croissanne/images that referenced this pull request Oct 28, 2025
@croissanne
osbuild/monitor: increase status scanner buffer size to 16MiB
b4027e6 
The size of the status message from the org.osbuild.rpm stage in the os
pipeline depends on how many packages a user has requested on top of the
base set. As a result, the monitor will always be breakable, but let's
increase the buffer size substantially to make it very unlikely that
users will hit this.

Especially with the addition of metadata [1], the status entry size grew
substantially for rpm stages. The rpm stage has an entry in the metadata
for each package that was installed (in addition to a very large output
message).

[1]: osbuild/osbuild#2215
croissanne added a commit to croissanne/images that referenced this pull request Oct 28, 2025
@croissanne
osbuild/monitor: increase status scanner buffer size to 16MiB
bd33d5d 
The size of the status message from the org.osbuild.rpm stage in the os
pipeline depends on how many packages a user has requested on top of the
base set. As a result, the monitor will always be breakable, but let's
increase the buffer size substantially to make it very unlikely that
users will hit this.

Especially with the addition of metadata [1], the status entry size grew
substantially for rpm stages. The rpm stage has an entry in the metadata
for each package that was installed (in addition to a very large output
message).

[1]: osbuild/osbuild#2215
github-merge-queue bot pushed a commit to osbuild/images that referenced this pull request Oct 28, 2025
@croissanne @mvo5
osbuild/monitor: increase status scanner buffer size to 16MiB
feb61ab 
The size of the status message from the org.osbuild.rpm stage in the os
pipeline depends on how many packages a user has requested on top of the
base set. As a result, the monitor will always be breakable, but let's
increase the buffer size substantially to make it very unlikely that
users will hit this.

Especially with the addition of metadata [1], the status entry size grew
substantially for rpm stages. The rpm stage has an entry in the metadata
for each package that was installed (in addition to a very large output
message).

[1]: osbuild/osbuild#2215
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@supakeen supakeen supakeen approved these changes

@lzap lzap Awaiting requested review from lzap lzap is a code owner automatically assigned from osbuild/osbuild-reviewers

@thozza thozza Awaiting requested review from thozza thozza is a code owner automatically assigned from osbuild/osbuild-reviewers

+1 more reviewer

@mvo5 mvo5 mvo5 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@croissanne @supakeen @mvo5