Skip to content

Add Log Handler Filter To workload tests to reduce logging noise#40700

Merged
bambriz merged 3 commits intoAzure:mainfrom
bambriz:diagnosticsfilterworkloadtest
Apr 24, 2025
Merged

Add Log Handler Filter To workload tests to reduce logging noise#40700
bambriz merged 3 commits intoAzure:mainfrom
bambriz:diagnosticsfilterworkloadtest

Conversation

@bambriz
Copy link
Copy Markdown
Member

@bambriz bambriz commented Apr 24, 2025

Description

Added filters for the workload test logger handler. It will only log when database account read happens, when latency is above 1000ms, and if there is an error. It won't log certain errors such as 404/0, 409/0, and 412/0. Given the test uses randint for upserting and reading documents a 404/0 and 409/0 is bound to occur and would just be considered noise for the workload tests.

The following filter is being used:

    def filter(self, record):
        # Check if the required attributes exists in the log record
        if all(hasattr(record, attr) for attr in _REQUIRED_ATTRIBUTES):
            # Check the conditions
            # Check database account reads
            if record.resource_type == "databaseaccount" and record.verb == "GET" and record.operation_type == "Read":
                return True
            # Check if there is an error and omit noisy errors
            if record.status_code >= 400 and not (
                    record.status_code in _NOISY_ERRORS and record.sub_status_code in _NOISY_SUB_STATUS_CODES):
                return True
            # Check if the latency (duration) was above 1000 ms
            if record.duration >= 1000:
                return True

        return False

Added filters for the worklaod test logger handler. It will only log when database account read happens, when latency is above 1000ms, and if there is an error. It won't log certain errors such as 404/0, 409/0, and 412/0. Given the test uses randint for upserting and reading documents a 404/0 and 409/0 is bound to occur and would just be considered noise for the workload tests.
Copilot AI review requested due to automatic review settings April 24, 2025 00:54
@bambriz bambriz requested a review from a team as a code owner April 24, 2025 00:54
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a filter to reduce logging noise during workload tests by filtering out non-critical log events.

  • Added definitions for _NOISY_ERRORS and _NOISY_SUB_STATUS_CODES.
  • Introduced the WorkloadLoggerFilter class with logic for filtering unnecessary log records.
  • Integrated the new filter into the logger configuration in create_logger.

@azure-sdk
Copy link
Copy Markdown
Collaborator

API change check

API changes are not detected in this pull request.

Copy link
Copy Markdown
Member

@tvaron3 tvaron3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Copy Markdown
Contributor

@allenkim0129 allenkim0129 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bambriz bambriz merged commit 5c9de2a into Azure:main Apr 24, 2025
18 checks passed
cRui861 pushed a commit that referenced this pull request May 14, 2025
)

* Update workload_utils.py

Added filters for the worklaod test logger handler. It will only log when database account read happens, when latency is above 1000ms, and if there is an error. It won't log certain errors such as 404/0, 409/0, and 412/0. Given the test uses randint for upserting and reading documents a 404/0 and 409/0 is bound to occur and would just be considered noise for the workload tests.

* Update workload_utils.py

* Update workload_utils.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Scale up test workload for Cosmos Python SDK for DR Drill and performance testing

5 participants