Skip to content

[ASM] Deactivate benchmark for legacy encoder to help CI#5299

Merged
anna-git merged 1 commit intomasterfrom
anna/asm/deactivate-benchmark-legacy-encoder
Mar 12, 2024
Merged

[ASM] Deactivate benchmark for legacy encoder to help CI#5299
anna-git merged 1 commit intomasterfrom
anna/asm/deactivate-benchmark-legacy-encoder

Conversation

@anna-git
Copy link
Contributor

@anna-git anna-git commented Mar 11, 2024 •
edited
Loading

Summary of changes

Deactivate benchmarks for legacy encoder for now, to relieve CI for now

Reason for change

Implementation details

Test coverage

Other details

@github-actions github-actions bot added area:tests unit tests, integration tests area:asm labels Mar 11, 2024
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Mar 11, 2024 •
edited
Loading

Datadog Report

Branch report: anna/asm/deactivate-benchmark-legacy-encoder
Commit report: ea0a0fc
Test service: dd-trace-dotnet

✅ 0 Failed, 336594 Passed, 1575 Skipped, 36m 19.18s Wall Time

@anna-git anna-git marked this pull request as ready for review March 11, 2024 16:28
@anna-git anna-git requested a review from a team as a code owner March 11, 2024 16:28
@andrewlock
Copy link
Member

andrewlock commented Mar 11, 2024

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5299) - mean (75ms)  : 66, 83
     .   : milestone, 75,
    master - mean (74ms)  : 65, 83
     .   : milestone, 74,

    section CallTarget+Inlining+NGEN
    This PR (5299) - mean (994ms)  : 974, 1014
     .   : milestone, 994,
    master - mean (985ms)  : 961, 1008
     .   : milestone, 985,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5299) - mean (110ms)  : 107, 113
     .   : milestone, 110,
    master - mean (118ms)  : 96, 140
     .   : milestone, 118,

    section CallTarget+Inlining+NGEN
    This PR (5299) - mean (717ms)  : 689, 746
     .   : milestone, 717,
    master - mean (713ms)  : 691, 735
     .   : milestone, 713,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5299) - mean (96ms)  : 91, 102
     .   : milestone, 96,
    master - mean (95ms)  : 91, 98
     .   : milestone, 95,

    section CallTarget+Inlining+NGEN
    This PR (5299) - mean (671ms)  : 648, 693
     .   : milestone, 671,
    master - mean (671ms)  : 647, 694
     .   : milestone, 671,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5299) - mean (188ms)  : 186, 191
     .   : milestone, 188,
    master - mean (188ms)  : 182, 193
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (5299) - mean (1,066ms)  : 1038, 1095
     .   : milestone, 1066,
    master - mean (1,063ms)  : 1038, 1089
     .   : milestone, 1063,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5299) - mean (271ms)  : 268, 274
     .   : milestone, 271,
    master - mean (269ms)  : 262, 276
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (5299) - mean (870ms)  : 844, 896
     .   : milestone, 870,
    master - mean (861ms)  : 839, 884
     .   : milestone, 861,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5299) - mean (260ms)  : 257, 264
     .   : milestone, 260,
    master - mean (258ms)  : 250, 265
     .   : milestone, 258,

    section CallTarget+Inlining+NGEN
    This PR (5299) - mean (853ms)  : 832, 874
     .   : milestone, 853,
    master - mean (855ms)  : 834, 876
     .   : milestone, 855,
Loading

@andrewlock
Copy link
Member

andrewlock commented Mar 12, 2024

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5299) (11.333M)   : 0, 11332815
    master (11.301M)   : 0, 11301171
    benchmarks/2.9.0 (11.048M)   : 0, 11047662

    section Automatic
    This PR (5299) (7.812M)   : 0, 7812170
    master (7.786M)   : 0, 7786387
    benchmarks/2.9.0 (7.876M)   : 0, 7876321

    section Trace stats
    This PR (5299) (8.169M)   : 0, 8169316
    master (8.162M)   : 0, 8161989

    section Manual
    This PR (5299) (9.996M)   : 0, 9996055
    master (9.926M)   : 0, 9926297

    section Manual + Automatic
    This PR (5299) (7.479M)   : 0, 7478602
    master (7.412M)   : 0, 7411648

    section Version Conflict
    This PR (5299) (6.754M)   : 0, 6753541
    master (6.730M)   : 0, 6730224
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5299) (9.628M)   : 0, 9627534
    master (9.529M)   : 0, 9529282
    benchmarks/2.9.0 (9.382M)   : 0, 9381950

    section Automatic
    This PR (5299) (6.646M)   : 0, 6645515
    master (6.506M)   : 0, 6505611

    section Trace stats
    This PR (5299) (6.855M)   : 0, 6855344
    master (6.940M)   : 0, 6940301

    section Manual
    This PR (5299) (8.247M)   : 0, 8246763
    master (8.383M)   : 0, 8383218

    section Manual + Automatic
    This PR (5299) (6.267M)   : 0, 6266632
    master (6.263M)   : 0, 6262794

    section Version Conflict
    This PR (5299) (5.722M)   : 0, 5721627
    master (5.698M)   : 0, 5698114
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5299) (9.307M)   : 0, 9306712
    master (10.221M)   : 0, 10221490
    benchmarks/2.9.0 (10.272M)   : 0, 10271759

    section Automatic
    This PR (5299) (6.659M)   : crit ,0, 6658792
    master (7.341M)   : 0, 7340809
    benchmarks/2.9.0 (7.644M)   : 0, 7644353

    section Trace stats
    This PR (5299) (7.041M)   : crit ,0, 7040756
    master (7.697M)   : 0, 7697428

    section Manual
    This PR (5299) (8.902M)   : 0, 8901554
    master (9.139M)   : 0, 9138836

    section Manual + Automatic
    This PR (5299) (7.001M)   : 0, 7001475
    master (7.059M)   : 0, 7058587

    section Version Conflict
    This PR (5299) (6.247M)   : 0, 6247479
    master (6.504M)   : 0, 6503694
Loading 
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.461M)   : 0, 7461396

    section No attack
    master (1.854M)   : 0, 1853890

    section Attack
    master (1.450M)   : 0, 1449724

    section Blocking
    master (3.161M)   : 0, 3160542

    section IAST default
    master (6.464M)   : 0, 6464188

    section IAST full
    master (5.600M)   : 0, 5600403

    section Base vuln
    master (0.918M)   : 0, 917723

    section IAST vuln
    master (0.859M)   : 0, 859223
Loading

@anna-git anna-git merged commit ea31cf5 into master Mar 12, 2024
@anna-git anna-git deleted the anna/asm/deactivate-benchmark-legacy-encoder branch March 12, 2024 15:39
@github-actions github-actions bot added this to the vNext milestone Mar 12, 2024
link04 added a commit that referenced this pull request Mar 12, 2024
@link04
Squashed commit of the following:
61e0fd2 
commit 832de4b
Author: Flavien Darche <11708575+e-n-0@users.noreply.github.com>
Date:   Tue Mar 12 20:24:21 2024 +0000

    [ASM][IAST] Configure maximum IAST Ranges (#5292)

    * Add configuration key

    * Use a RangeList in some case to not exceed the max number

    * Revert some code + implem correct merge

    * Fix + Add unit and integration tests

    * Usual macos fix for snapshot

    * Fix snapshots hashs

    * Update snapshots (remove other tests as they can't apply different env var values in same run)

    * Apply comment

    * Re-integrate integration tests with multiple processes (new fixture)

    * Add test case for setting MaxRangeCount to zero

commit 83f6ab1
Author: Tony Redondo <tony.redondo@datadoghq.com>
Date:   Tue Mar 12 21:20:39 2024 +0100

    [CI Visibility] - Enable snapshot testing of current testing framework implementations (#5226)

commit 233695a
Author: Daniel Romano <108014683+daniel-romano-DD@users.noreply.github.com>
Date:   Tue Mar 12 17:06:06 2024 +0100

    [IAST] Vulnerability and Evidence truncation (#5302)

    * Initial implementation

    * Updated test bundle

    * Fix test compilation error

    * Fix snapshot (from rebase)

    * Fix typo in config value. Updated tests

    * Fix typo

    * Refactor converters initialization

commit ea31cf5
Author: Anna <anna.yafi@datadoghq.com>
Date:   Tue Mar 12 16:39:09 2024 +0100

    Deactivate benchmark for legacy encoder (#5299)

commit d0d713a
Author: NachoEchevarria <53266532+NachoEchevarria@users.noreply.github.com>
Date:   Tue Mar 12 09:25:27 2024 +0100

    Set big regex timeouts for tests (#5297)

commit d5388d6
Author: Lucas Pimentel <lucas.pimentel@datadoghq.com>
Date:   Mon Mar 11 15:20:58 2024 -0400

    [Tracing] Support configuring `DD_TRACE_ENABLED` remotely (#5181)

    * add support for remote TraceEnabled setting

    * fix unrelated typo

    * add ApmTracingEnabled capability 19

    * add missing RCM capability 18

    * add mapping

    * add unit test

    * add comments to unit test

    * rename property to match RCM constant

    * include config in integration tests

    * fix test json

    * rewrite tests to use raw values instead of strings

commit 2b95f46
Author: Flavien Darche <11708575+e-n-0@users.noreply.github.com>
Date:   Mon Mar 11 17:47:55 2024 +0100

    [ASM][IAST] Support manual JSON deserialisation (Newtonsoft.Json) (#5238)

    * Add Newtonsoft.Json (non -working yet)

    * Refactor the tainting proces + add tests

    * Add the JToken Parse aspect + test

    * Rename Aspects class + Duck orignal method call

    * Add integration test

    * Fix nullability

    * Fix compilation issue for netfx

    * Change JSON formatting in ParseTests

    * Fix a test json format

    * Refactor NewtonsoftJsonAspects to static constructor

commit 0d511f9
Author: Igor Kravchenko <21974069+kr-igor@users.noreply.github.com>
Date:   Mon Mar 11 09:35:23 2024 -0500

    [DSM] - Fixes for IbmMq instrumentation (#5271)

    * Use byte properties instead of strings

    * Fixed nullability files

    * Added some debug info

    * Fixed lint issues

    * Added a bit more logs

    * Using slow byte->sbyte conversion

    * Added noop headers adapter

    * Fixed nullability files

    * Added more logs

    * Cleaned up debug logs

    * Removed symlink

    * Update tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/IbmMq/IbmMqHeadersAdapter.cs

    Removed debug code

    Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>

    * Update tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/IbmMq/IbmMqHeadersAdapter.cs

    Using Unsafe.As instead of BlockCopy

    Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>

    * Update tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/IbmMq/IbmMqHeadersAdapter.cs

    Use Unsafe.As instead of BlockCopy

    Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>

    * Addressed some of the comments

    * Removed context propagation options

    ---------

    Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>

commit 5684a72
Author: Zach Montoya <zach.montoya@datadoghq.com>
Date:   Fri Mar 8 20:56:30 2024 -0800

    [Tracing] Update instrumentation point for DD_TRACE_DELAY_WCF_INSTRUMENTATION_ENABLED=true (#5206)

    Updates the instrumentation point for `DD_TRACE_DELAY_WCF_INSTRUMENTATION_ENABLED=true` so that now a server span is created immediately before IDispatchMessageInspector implementations are run, so application code can access the root span from inside a IDispatchMessageInspector.AfterReceiveRequest callback.

    This PR also does some cleanup to remove unused Wcf files and it makes the entire Wcf instrumentation use nullable reference types.

commit ca1bb6e
Author: Andrew Lock <andrew.lock@datadoghq.com>
Date:   Fri Mar 8 17:43:57 2024 +0000

    Fix errors identified from telemetry (#5279)

    * Try to avoid MongoDb exception

    We're seeing exceptions like this:
    ```
    System.FieldAccessException
       at REDACTED
       at Datadog.Trace.ClrProfiler.AutoInstrumentation.MongoDb.MongoDbIntegration.CreateScope[TConnection](Object wireProtocol, TConnection connection)
       at REDACTED
       at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken)
    ```

    and the only explanation I can think of is a duck-chaining issue, so stopped doing duck chaining and being explicit instead

    * Add local functions to try to isolate problems

    * Fix ArgumentNullException in AWS SQS integration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@robertpi robertpi robertpi approved these changes

@e-n-0 e-n-0 e-n-0 approved these changes

Assignees

No one assigned

Labels

area:asm area:tests unit tests, integration tests

Projects

None yet

Milestone

2.49.0

Development

Successfully merging this pull request may close these issues.

4 participants

@anna-git @andrewlock @robertpi @e-n-0