[Tests] Add pod log collection for model monitoring test failures #9000
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add automatic collection and logging of pod logs when model monitoring system tests fail, to help debug CI failures. Changes: - Add collect_monitoring_pod_logs() to TestMLRunSystemModelMonitoring - Collect logs from monitoring pods (stream, controller, writer, serving) - Collect filtered error logs from mlrun-api pods mentioning test project - Add pytest hook in conftest.py to trigger on test failure - Requires MLRUN_SYSTEM_TEST_KUBECONFIG_PATH env var to enable ML-11480 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
liranbg
reviewed
Dec 1, 2025
Member
liranbg
left a comment
There was a problem hiding this comment.
I think this can be more generalized making rest of the test suites dump their logs upon error. overall, you could collect all system pods with certain labels (the ones mlrun set across) and that would be helpful when debugging any failing test
Member
Yep, I second that! |
Move pod log collection from model_monitoring-specific to all system tests: - Add collect_pod_logs_on_failure() to TestMLRunSystem base class - Project pods (name contains project_name): collect full logs - System pods (mlrun-api-*): collect time-bounded logs via since_seconds - Add autouse fixture to track test start time for duration calculation - Remove duplicate code from model_monitoring/__init__.py - Delete model_monitoring/conftest.py (functionality moved to system level) This addresses reviewer feedback to make pod log collection available for debugging failures in any system test suite, not just model monitoring.
assaf758
approved these changes
Dec 2, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add automatic pod log collection when model monitoring system tests fail to help debug CI failures.
Changes Made
collect_monitoring_pod_logs()and helper methods toTestMLRunSystemModelMonitoringconftest.pyto trigger log collection on test failureTesting
Reference