Skip to content

Unexpected behaviour with sampling overrides config #2152

@umutesen

Description

@umutesen

Expected behavior

I expect telemetry spans to be ignored when using sampling overrides.

Actual behavior

It appears telemetry spans that do not match sampling overrides criteria are still exported and sent to the ingestion service.

We have a web application (Angular) which automatically tracks AJAX dependencies.

Our API is Java based, built with Spring Boot. There are other APIs running in the same JVM alongside of our API, which we do not want to track with Application Insights.

Both Angular and Java Agent share the same instrumentation key with CORS correlation enabled in web client.

This behaviour only applicable when AJAX request initiates the operation. Telemetry is not collected when hitting another-api directly from Postman. Also, ignoring the AJAX dependency that initiates the operation in JavaScript makes no difference, server-side telemetry is still collected when using the browser.

To Reproduce

Consider the following Java agent config, where we default sampling rate to 0 and use sampling overrides to only capture telemetry for traffic into /my-api/v1.

When web client makes an AJAX request to /another-api/v2 running on the same JVM, all remote dependency is captured and ingested.

I was only expecting to see the AJAX dependency from the client and nothing from the server due to to my sampling overrides configuration.

java agent config:

{
  "connectionString": "InstrumentationKey=XXXX",
  "role": {
    "name": "My.API"
  },
  "sampling": {
    "percentage": 0
  },
  "preview": {
    "sampling": {
      "overrides": [
        {
          "attributes": [
            {
              "key": "http.url",
              "value": "\/my-api\/v1\/",
              "matchType": "regexp"
            },
            {
              "key": "http.method",
              "value": "^(GET|POST|PUT|DELETE)",
              "matchType": "regexp"
            }
          ],
          "percentage": 100
        }
      ]
    },
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "regexp",
          "metricNames": [
            ".*"
          ]
        }
      }
    ]
  },
  "instrumentation": {
    "logging": {
      "level": "ERROR"
    }
  },
  "selfDiagnostics": {
    "destination": "file",
    "level": "TRACE",
    "file": {
      "path": "applicationinsights.log",
      "maxSizeMb": 5,
      "maxHistory": 1
    }
  }
}

System information

Please provide the following information:

  • SDK Version: 3.2.7
  • OS type and version: Windows 10
  • Application Server type and version (if applicable): JBoss EAP 6.1
  • Using spring-boot? Yes

Logs

Here is one of the remote dependencies captured as part of calling another-api/v2

2022-03-01 14:43:32.850Z DEBUG c.m.a.a.internal.exporter.Exporter - exporting span: SpanData{spanContext=ImmutableSpanContext{traceId=099ac4d01412484fa5697d0389146c65, spanId=bfe7a6d468d8ee10, traceFlags=01, traceState=ArrayBasedTraceState{entries=[ai-internal-sp, 0]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=099ac4d01412484fa5697d0389146c65, spanId=e48d0528452b22cc, traceFlags=01, traceState=ArrayBasedTraceState{entries=[ai-internal-sp, 0]}, remote=false, valid=true}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.8.0, attributes={host.arch="amd64", host.name="XXXX", os.description="Windows 10 10.0", os.type="windows", process.command_line="C:\Program Files\Zulu\zulu-8\jre;bin;java.exe -Dprogram.name=JBossTools: JBoss Enterprise Application Platform 6.1+ -Xms1024m -Xmx1024m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Djava.net.preferIPv4Stack=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Dorg.jboss.boot.log.file=C:\Development\svn\JBoss\jboss-eap-6.4\standalone\log\boot.log -Dlogging.configuration=file:C:\Development\svn\JBoss\jboss-eap-6.4\standalone\configuration\logging.properties -Djboss.home.dir=C:\Development\svn\JBoss\jboss-eap-6.4 -Dorg.jboss.logmanager.nocolor=true -Djboss.bind.address.management=localhost -Duse.local.dashboard=true -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar -Djboss.modules.system.pkgs=org.jboss.byteman,com.sun.crypto.provider,org.jboss.logmanager -javaagent:C:\Users\XXX\Downloads\applicationinsights-agent-3.2.7.jar -Dfile.encoding=Cp1252 -Xbootclasspath:C:\Program Files\Zulu\zulu-8\jre\lib\resources.jar;C:\Program Files\Zulu\zulu-8\jre\lib\rt.jar;C:\Program Files\Zulu\zulu-8\jre\lib\jsse.jar;C:\Program Files\Zulu\zulu-8\jre\lib\jce.jar;C:\Program Files\Zulu\zulu-8\jre\lib\charsets.jar;C:\Program Files\Zulu\zulu-8\jre\lib\jfr.jar", process.executable.path="C:\Program Files\Zulu\zulu-8\jre;bin;java.exe", process.pid=54460, process.runtime.description="Azul Systems, Inc. OpenJDK 64-Bit Server VM 25.311-b02", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="1.8.0_311-b02", service.name="My.API", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.10.0"}}, instrumentationLibraryInfo=InstrumentationLibraryInfo{name=io.opentelemetry.jdbc, version=null, schemaUrl=null}, name=SELECT XXX, kind=CLIENT, startEpochNanos=1646145812849171800,  endEpochNanos=1646145812850286800, attributes=AttributesMap{data={db.name=XXX, thread.name=http-localhost/127.0.0.1:8080-4,  applicationinsights.internal.operation_name=GET /another-api/v2/XXX, db.operation=SELECT, net.peer.port=1433, db.system=mssql,  db.statement=select ? XXX, thread.id=396, db.connection_string=sqlserver://XX.xx.test:1433, net.peer.name=XX.xx.test}, capacity=128, totalAddedValues=10}, totalAttributeCount=10, events=[],  totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions