feat(ci_visibility): bazel offline cache and payload-file modes#17197
Conversation
Add constants, resolve_rlocation utility, OfflineMode singleton, and NoOpBackendConnectorSetup for Bazel hermetic sandbox support. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t mode When DD_TEST_OPTIMIZATION_MANIFEST_FILE is set (Bazel sandbox), all four API calls (settings, known_tests, test_management, skippable_tests) now read from pre-fetched JSON files in .testoptimization/cache/http/ instead of making HTTP requests. SessionManager uses NoOpBackendConnectorSetup to avoid network setup failures inside the sandbox. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e modes In manifest mode, git subprocess calls and pack file uploads are skipped since the external tool has already handled git data. In payload-files mode, CI/Git/OS/runtime tags are omitted from payloads so the external tool can enrich them from context.json without stale values invalidating Bazel's cache. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES is true, TestOptWriter writes test event payloads as JSON files to TEST_UNDECLARED_OUTPUTS_DIR/payloads/ tests/ and TestCoverageWriter writes coverage to payloads/coverage/. Files are written atomically (write + rename). Coverage report HTTP upload is skipped since the file-based path already captures coverage data. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…Bazel file reads Replace per-method offline branches inside APIClient with a strategy pattern: a TestOptDataProvider Protocol is satisfied by both APIClient (HTTP) and the new CachedFileDataProvider (reads from .testoptimization/ cache/http/). SessionManager picks the implementation at construction time based on manifest mode, keeping APIClient focused purely on HTTP. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Performance SLOsComparing candidate gnufede/SDTEST-3334-bazel-pr4-payload-files (a08f1af) with baseline main (97d4f14) 📈 Performance Regressions (2 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 104.010µs (SLO: <130.000µs 📉 -20.0%) vs baseline: +4.5% Memory: ✅ 43.967MB (SLO: <46.000MB -4.4%) vs baseline: +5.1% ✅ add_inplace_aspectTime: ✅ 102.106µs (SLO: <130.000µs 📉 -21.5%) vs baseline: +1.3% Memory: ✅ 43.925MB (SLO: <46.000MB -4.5%) vs baseline: +4.8% ✅ add_inplace_noaspectTime: ✅ 28.298µs (SLO: <40.000µs 📉 -29.3%) vs baseline: +0.3% Memory: ✅ 44.109MB (SLO: <46.000MB -4.1%) vs baseline: +5.2% ✅ add_noaspectTime: ✅ 48.587µs (SLO: <70.000µs 📉 -30.6%) vs baseline: -1.1% Memory: ✅ 44.077MB (SLO: <46.000MB -4.2%) vs baseline: +5.3% ✅ bytearray_aspectTime: ✅ 250.337µs (SLO: <400.000µs 📉 -37.4%) vs baseline: ~same Memory: ✅ 44.050MB (SLO: <46.000MB -4.2%) vs baseline: +5.2% ✅ bytearray_extend_aspectTime: ✅ 652.872µs (SLO: <800.000µs 📉 -18.4%) vs baseline: ~same Memory: ✅ 43.966MB (SLO: <46.000MB -4.4%) vs baseline: +5.3% ✅ bytearray_extend_noaspectTime: ✅ 265.010µs (SLO: <400.000µs 📉 -33.7%) vs baseline: ~same Memory: ✅ 44.372MB (SLO: <46.000MB -3.5%) vs baseline: +6.0% ✅ bytearray_noaspectTime: ✅ 140.840µs (SLO: <300.000µs 📉 -53.1%) vs baseline: +1.9% Memory: ✅ 44.175MB (SLO: <46.000MB -4.0%) vs baseline: +5.6% ✅ bytes_aspectTime: ✅ 219.250µs (SLO: <300.000µs 📉 -26.9%) vs baseline: +0.8% Memory: ✅ 43.982MB (SLO: <46.000MB -4.4%) vs baseline: +4.9% ✅ bytes_noaspectTime: ✅ 133.986µs (SLO: <200.000µs 📉 -33.0%) vs baseline: +1.3% Memory: ✅ 43.941MB (SLO: <46.000MB -4.5%) vs baseline: +4.8% ✅ bytesio_aspectTime: ✅ 3.783ms (SLO: <5.000ms 📉 -24.3%) vs baseline: +0.9% Memory: ✅ 43.864MB (SLO: <46.000MB -4.6%) vs baseline: +4.6% ✅ bytesio_noaspectTime: ✅ 314.373µs (SLO: <420.000µs 📉 -25.1%) vs baseline: -0.6% Memory: ✅ 44.099MB (SLO: <46.000MB -4.1%) vs baseline: +5.4% ✅ capitalize_aspectTime: ✅ 87.956µs (SLO: <300.000µs 📉 -70.7%) vs baseline: +0.4% Memory: ✅ 44.047MB (SLO: <46.000MB -4.2%) vs baseline: +5.0% ✅ capitalize_noaspectTime: ✅ 250.057µs (SLO: <300.000µs 📉 -16.6%) vs baseline: +2.0% Memory: ✅ 44.081MB (SLO: <46.000MB -4.2%) vs baseline: +5.3% ✅ casefold_aspectTime: ✅ 88.315µs (SLO: <500.000µs 📉 -82.3%) vs baseline: -0.2% Memory: ✅ 44.094MB (SLO: <46.000MB -4.1%) vs baseline: +5.2% ✅ casefold_noaspectTime: ✅ 304.283µs (SLO: <500.000µs 📉 -39.1%) vs baseline: +0.4% Memory: ✅ 44.080MB (SLO: <46.000MB -4.2%) vs baseline: +4.9% ✅ decode_aspectTime: ✅ 86.693µs (SLO: <100.000µs 📉 -13.3%) vs baseline: +0.4% Memory: ✅ 44.129MB (SLO: <46.000MB -4.1%) vs baseline: +5.2% ✅ decode_noaspectTime: ✅ 151.508µs (SLO: <210.000µs 📉 -27.9%) vs baseline: -1.8% Memory: ✅ 44.013MB (SLO: <46.000MB -4.3%) vs baseline: +4.9% ✅ encode_aspectTime: ✅ 84.083µs (SLO: <200.000µs 📉 -58.0%) vs baseline: -0.3% Memory: ✅ 44.070MB (SLO: <46.000MB -4.2%) vs baseline: +5.2% ✅ encode_noaspectTime: ✅ 140.579µs (SLO: <200.000µs 📉 -29.7%) vs baseline: -1.5% Memory: ✅ 43.959MB (SLO: <46.000MB -4.4%) vs baseline: +5.1% ✅ format_aspectTime: ✅ 14.616ms (SLO: <19.200ms 📉 -23.9%) vs baseline: ~same Memory: ✅ 44.388MB (SLO: <46.000MB -3.5%) vs baseline: +5.4% ✅ format_map_aspectTime: ✅ 16.383ms (SLO: <21.500ms 📉 -23.8%) vs baseline: -0.2% Memory: ✅ 44.409MB (SLO: <46.000MB -3.5%) vs baseline: +5.6% ✅ format_map_noaspectTime: ✅ 372.265µs (SLO: <500.000µs 📉 -25.5%) vs baseline: +0.3% Memory: ✅ 44.095MB (SLO: <46.000MB -4.1%) vs baseline: +5.5% ✅ format_noaspectTime: ✅ 301.346µs (SLO: <500.000µs 📉 -39.7%) vs baseline: -0.9% Memory: ✅ 44.026MB (SLO: <46.000MB -4.3%) vs baseline: +5.0% ✅ index_aspectTime: ✅ 138.029µs (SLO: <300.000µs 📉 -54.0%) vs baseline: 📈 +12.9% Memory: ✅ 44.156MB (SLO: <46.000MB -4.0%) vs baseline: +5.4% ✅ index_noaspectTime: ✅ 40.336µs (SLO: <300.000µs 📉 -86.6%) vs baseline: -0.3% Memory: ✅ 43.975MB (SLO: <46.000MB -4.4%) vs baseline: +5.0% ✅ join_aspectTime: ✅ 210.464µs (SLO: <300.000µs 📉 -29.8%) vs baseline: -0.2% Memory: ✅ 44.051MB (SLO: <46.000MB -4.2%) vs baseline: +5.5% ✅ join_noaspectTime: ✅ 142.006µs (SLO: <300.000µs 📉 -52.7%) vs baseline: ~same Memory: ✅ 44.194MB (SLO: <46.000MB -3.9%) vs baseline: +5.4% ✅ ljust_aspectTime: ✅ 503.758µs (SLO: <700.000µs 📉 -28.0%) vs baseline: +0.2% Memory: ✅ 44.022MB (SLO: <46.000MB -4.3%) vs baseline: +5.0% ✅ ljust_noaspectTime: ✅ 269.332µs (SLO: <300.000µs 📉 -10.2%) vs baseline: +5.2% Memory: ✅ 44.082MB (SLO: <46.000MB -4.2%) vs baseline: +5.0% ✅ lower_aspectTime: ✅ 294.530µs (SLO: <500.000µs 📉 -41.1%) vs baseline: -1.4% Memory: ✅ 44.118MB (SLO: <46.000MB -4.1%) vs baseline: +5.6% ✅ lower_noaspectTime: ✅ 232.568µs (SLO: <300.000µs 📉 -22.5%) vs baseline: +0.1% Memory: ✅ 44.036MB (SLO: <46.000MB -4.3%) vs baseline: +5.4% ✅ lstrip_aspectTime: ✅ 0.274ms (SLO: <3.000ms 📉 -90.9%) vs baseline: -0.7% Memory: ✅ 44.223MB (SLO: <46.000MB -3.9%) vs baseline: +5.4% ✅ lstrip_noaspectTime: ✅ 0.176ms (SLO: <3.000ms 📉 -94.1%) vs baseline: -1.0% Memory: ✅ 43.913MB (SLO: <46.000MB -4.5%) vs baseline: +4.7% ✅ modulo_aspectTime: ✅ 14.274ms (SLO: <18.750ms 📉 -23.9%) vs baseline: ~same Memory: ✅ 44.222MB (SLO: <46.000MB -3.9%) vs baseline: +5.1% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 14.779ms (SLO: <19.350ms 📉 -23.6%) vs baseline: ~same Memory: ✅ 44.290MB (SLO: <46.000MB -3.7%) vs baseline: +5.3% ✅ modulo_aspect_for_bytesTime: ✅ 14.380ms (SLO: <18.900ms 📉 -23.9%) vs baseline: -0.3% Memory: ✅ 44.245MB (SLO: <46.000MB -3.8%) vs baseline: +5.4% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 14.673ms (SLO: <19.150ms 📉 -23.4%) vs baseline: +0.6% Memory: ✅ 44.183MB (SLO: <46.000MB -3.9%) vs baseline: +5.0% ✅ modulo_noaspectTime: ✅ 0.353ms (SLO: <3.000ms 📉 -88.2%) vs baseline: -0.3% Memory: ✅ 43.992MB (SLO: <46.000MB -4.4%) vs baseline: +5.1% ✅ replace_aspectTime: ✅ 18.410ms (SLO: <24.000ms 📉 -23.3%) vs baseline: +0.2% Memory: ✅ 44.303MB (SLO: <46.000MB -3.7%) vs baseline: +5.3% ✅ replace_noaspectTime: ✅ 280.298µs (SLO: <400.000µs 📉 -29.9%) vs baseline: -0.8% Memory: ✅ 44.042MB (SLO: <46.000MB -4.3%) vs baseline: +5.3% ✅ repr_aspectTime: ✅ 320.104µs (SLO: <420.000µs 📉 -23.8%) vs baseline: +0.5% Memory: ✅ 44.189MB (SLO: <46.000MB -3.9%) vs baseline: +5.6% ✅ repr_noaspectTime: ✅ 46.761µs (SLO: <90.000µs 📉 -48.0%) vs baseline: -0.5% Memory: ✅ 43.883MB (SLO: <46.000MB -4.6%) vs baseline: +5.0% ✅ rstrip_aspectTime: ✅ 384.998µs (SLO: <500.000µs 📉 -23.0%) vs baseline: ~same Memory: ✅ 44.117MB (SLO: <46.000MB -4.1%) vs baseline: +5.2% ✅ rstrip_noaspectTime: ✅ 181.575µs (SLO: <300.000µs 📉 -39.5%) vs baseline: +0.9% Memory: ✅ 44.116MB (SLO: <46.000MB -4.1%) vs baseline: +5.5% ✅ slice_aspectTime: ✅ 185.285µs (SLO: <300.000µs 📉 -38.2%) vs baseline: +0.3% Memory: ✅ 44.013MB (SLO: <46.000MB -4.3%) vs baseline: +4.8% ✅ slice_noaspectTime: ✅ 54.451µs (SLO: <90.000µs 📉 -39.5%) vs baseline: -0.7% Memory: ✅ 44.026MB (SLO: <46.000MB -4.3%) vs baseline: +5.1% ✅ stringio_aspectTime: ✅ 4.393ms (SLO: <5.000ms 📉 -12.1%) vs baseline: 📈 +16.0% Memory: ✅ 43.931MB (SLO: <46.000MB -4.5%) vs baseline: +5.0% ✅ stringio_noaspectTime: ✅ 345.208µs (SLO: <500.000µs 📉 -31.0%) vs baseline: -0.6% Memory: ✅ 44.074MB (SLO: <46.000MB -4.2%) vs baseline: +4.9% ✅ strip_aspectTime: ✅ 275.577µs (SLO: <350.000µs 📉 -21.3%) vs baseline: +1.1% Memory: ✅ 43.978MB (SLO: <46.000MB -4.4%) vs baseline: +5.1% ✅ strip_noaspectTime: ✅ 176.593µs (SLO: <240.000µs 📉 -26.4%) vs baseline: +0.3% Memory: ✅ 44.035MB (SLO: <46.000MB -4.3%) vs baseline: +5.2% ✅ swapcase_aspectTime: ✅ 331.181µs (SLO: <500.000µs 📉 -33.8%) vs baseline: ~same Memory: ✅ 44.046MB (SLO: <46.000MB -4.2%) vs baseline: +5.1% ✅ swapcase_noaspectTime: ✅ 268.814µs (SLO: <400.000µs 📉 -32.8%) vs baseline: ~same Memory: ✅ 43.922MB (SLO: <46.000MB -4.5%) vs baseline: +4.6% ✅ title_aspectTime: ✅ 317.929µs (SLO: <500.000µs 📉 -36.4%) vs baseline: -1.3% Memory: ✅ 44.152MB (SLO: <46.000MB -4.0%) vs baseline: +5.4% ✅ title_noaspectTime: ✅ 256.852µs (SLO: <400.000µs 📉 -35.8%) vs baseline: +0.2% Memory: ✅ 43.999MB (SLO: <46.000MB -4.4%) vs baseline: +4.9% ✅ translate_aspectTime: ✅ 487.567µs (SLO: <700.000µs 📉 -30.3%) vs baseline: -0.8% Memory: ✅ 44.018MB (SLO: <46.000MB -4.3%) vs baseline: +4.9% ✅ translate_noaspectTime: ✅ 424.867µs (SLO: <500.000µs 📉 -15.0%) vs baseline: -0.2% Memory: ✅ 44.014MB (SLO: <46.000MB -4.3%) vs baseline: +5.2% ✅ upper_aspectTime: ✅ 295.599µs (SLO: <500.000µs 📉 -40.9%) vs baseline: ~same Memory: ✅ 44.032MB (SLO: <46.000MB -4.3%) vs baseline: +5.2% ✅ upper_noaspectTime: ✅ 232.496µs (SLO: <400.000µs 📉 -41.9%) vs baseline: +0.9% Memory: ✅ 44.074MB (SLO: <46.000MB -4.2%) vs baseline: +5.4% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 509.969µs (SLO: <700.000µs 📉 -27.1%) vs baseline: 📈 +21.1% Memory: ✅ 43.876MB (SLO: <46.000MB -4.6%) vs baseline: +4.9% ✅ ospathbasename_noaspectTime: ✅ 431.477µs (SLO: <700.000µs 📉 -38.4%) vs baseline: -0.6% Memory: ✅ 43.979MB (SLO: <46.000MB -4.4%) vs baseline: +5.2% ✅ ospathjoin_aspectTime: ✅ 626.810µs (SLO: <700.000µs 📉 -10.5%) vs baseline: +0.6% Memory: ✅ 43.842MB (SLO: <46.000MB -4.7%) vs baseline: +4.9% ✅ ospathjoin_noaspectTime: ✅ 638.414µs (SLO: <700.000µs -8.8%) vs baseline: +0.6% Memory: ✅ 43.800MB (SLO: <46.000MB -4.8%) vs baseline: +4.7% ✅ ospathnormcase_aspectTime: ✅ 350.948µs (SLO: <700.000µs 📉 -49.9%) vs baseline: -0.1% Memory: ✅ 43.875MB (SLO: <46.000MB -4.6%) vs baseline: +5.1% ✅ ospathnormcase_noaspectTime: ✅ 359.286µs (SLO: <700.000µs 📉 -48.7%) vs baseline: +0.8% Memory: ✅ 43.873MB (SLO: <46.000MB -4.6%) vs baseline: +4.9% ✅ ospathsplit_aspectTime: ✅ 478.366µs (SLO: <700.000µs 📉 -31.7%) vs baseline: -0.7% Memory: ✅ 44.031MB (SLO: <46.000MB -4.3%) vs baseline: +5.1% ✅ ospathsplit_noaspectTime: ✅ 492.100µs (SLO: <700.000µs 📉 -29.7%) vs baseline: -0.5% Memory: ✅ 43.884MB (SLO: <46.000MB -4.6%) vs baseline: +4.8% ✅ ospathsplitdrive_aspectTime: ✅ 372.905µs (SLO: <700.000µs 📉 -46.7%) vs baseline: -0.8% Memory: ✅ 43.833MB (SLO: <46.000MB -4.7%) vs baseline: +4.6% ✅ ospathsplitdrive_noaspectTime: ✅ 72.347µs (SLO: <700.000µs 📉 -89.7%) vs baseline: -1.0% Memory: ✅ 43.918MB (SLO: <46.000MB -4.5%) vs baseline: +5.1% ✅ ospathsplitext_aspectTime: ✅ 457.010µs (SLO: <700.000µs 📉 -34.7%) vs baseline: -0.5% Memory: ✅ 43.934MB (SLO: <46.000MB -4.5%) vs baseline: +5.1% ✅ ospathsplitext_noaspectTime: ✅ 464.045µs (SLO: <700.000µs 📉 -33.7%) vs baseline: ~same Memory: ✅ 43.975MB (SLO: <46.000MB -4.4%) vs baseline: +5.4% 🟡 Near SLO Breach (3 suites)🟡 djangosimple - 30/30✅ appsecTime: ✅ 19.739ms (SLO: <22.300ms 📉 -11.5%) vs baseline: +0.1% Memory: ✅ 69.335MB (SLO: <73.500MB -5.7%) vs baseline: +4.8% ✅ exception-replay-enabledTime: ✅ 1.322ms (SLO: <1.450ms -8.8%) vs baseline: -0.3% Memory: ✅ 67.522MB (SLO: <71.500MB -5.6%) vs baseline: +4.9% ✅ iastTime: ✅ 19.844ms (SLO: <22.250ms 📉 -10.8%) vs baseline: +0.8% Memory: ✅ 69.304MB (SLO: <75.000MB -7.6%) vs baseline: +4.9% ✅ profilerTime: ✅ 15.102ms (SLO: <16.550ms -8.7%) vs baseline: -0.3% Memory: ✅ 60.481MB (SLO: <61.000MB 🟡 -0.9%) vs baseline: +4.8% ✅ resource-renamingTime: ✅ 19.600ms (SLO: <21.750ms -9.9%) vs baseline: ~same Memory: ✅ 69.341MB (SLO: <73.500MB -5.7%) vs baseline: +4.8% ✅ span-code-originTime: ✅ 20.240ms (SLO: <28.200ms 📉 -28.2%) vs baseline: +1.7% Memory: ✅ 69.592MB (SLO: <75.000MB -7.2%) vs baseline: +4.9% ✅ tracerTime: ✅ 19.876ms (SLO: <21.750ms -8.6%) vs baseline: +1.1% Memory: ✅ 69.302MB (SLO: <75.000MB -7.6%) vs baseline: +4.8% ✅ tracer-and-profilerTime: ✅ 21.154ms (SLO: <23.500ms -10.0%) vs baseline: +0.1% Memory: ✅ 71.301MB (SLO: <75.000MB -4.9%) vs baseline: +4.6% ✅ tracer-dont-create-db-spansTime: ✅ 19.797ms (SLO: <21.500ms -7.9%) vs baseline: +0.6% Memory: ✅ 69.332MB (SLO: <75.000MB -7.6%) vs baseline: +4.8% ✅ tracer-minimalTime: ✅ 16.955ms (SLO: <17.500ms -3.1%) vs baseline: +0.5% Memory: ✅ 69.326MB (SLO: <75.000MB -7.6%) vs baseline: +4.9% ✅ tracer-nativeTime: ✅ 19.726ms (SLO: <21.750ms -9.3%) vs baseline: +0.1% Memory: ✅ 69.390MB (SLO: <72.500MB -4.3%) vs baseline: +4.9% ✅ tracer-no-cachesTime: ✅ 17.635ms (SLO: <19.650ms 📉 -10.3%) vs baseline: -0.1% Memory: ✅ 69.359MB (SLO: <75.000MB -7.5%) vs baseline: +4.9% ✅ tracer-no-databasesTime: ✅ 19.425ms (SLO: <20.100ms -3.4%) vs baseline: +0.6% Memory: ✅ 69.390MB (SLO: <75.000MB -7.5%) vs baseline: +5.0% ✅ tracer-no-middlewareTime: ✅ 19.407ms (SLO: <21.500ms -9.7%) vs baseline: -0.2% Memory: ✅ 69.286MB (SLO: <75.000MB -7.6%) vs baseline: +4.7% ✅ tracer-no-templatesTime: ✅ 19.583ms (SLO: <22.000ms 📉 -11.0%) vs baseline: +0.3% Memory: ✅ 69.339MB (SLO: <73.500MB -5.7%) vs baseline: +5.1% 🟡 flasksimple - 18/18✅ appsec-getTime: ✅ 3.377ms (SLO: <4.750ms 📉 -28.9%) vs baseline: +0.2% Memory: ✅ 56.558MB (SLO: <66.500MB 📉 -15.0%) vs baseline: +4.8% ✅ appsec-postTime: ✅ 2.866ms (SLO: <6.750ms 📉 -57.5%) vs baseline: +0.1% Memory: ✅ 56.567MB (SLO: <66.500MB 📉 -14.9%) vs baseline: +4.8% ✅ appsec-telemetryTime: ✅ 3.385ms (SLO: <4.750ms 📉 -28.7%) vs baseline: +1.1% Memory: ✅ 56.609MB (SLO: <66.500MB 📉 -14.9%) vs baseline: +5.0% ✅ debuggerTime: ✅ 1.879ms (SLO: <2.000ms -6.1%) vs baseline: -0.2% Memory: ✅ 49.258MB (SLO: <51.500MB -4.4%) vs baseline: +4.8% ✅ iast-getTime: ✅ 1.875ms (SLO: <2.000ms -6.2%) vs baseline: +0.1% Memory: ✅ 45.967MB (SLO: <49.000MB -6.2%) vs baseline: +4.9% ✅ profilerTime: ✅ 1.908ms (SLO: <2.100ms -9.1%) vs baseline: -0.2% Memory: ✅ 52.583MB (SLO: <53.500MB 🟡 -1.7%) vs baseline: +5.0% ✅ resource-renamingTime: ✅ 3.350ms (SLO: <3.650ms -8.2%) vs baseline: ~same Memory: ✅ 56.471MB (SLO: <60.000MB -5.9%) vs baseline: +4.8% ✅ tracerTime: ✅ 3.363ms (SLO: <3.650ms -7.9%) vs baseline: +0.1% Memory: ✅ 56.516MB (SLO: <60.000MB -5.8%) vs baseline: +4.7% ✅ tracer-nativeTime: ✅ 3.352ms (SLO: <3.650ms -8.2%) vs baseline: ~same Memory: ✅ 56.404MB (SLO: <60.000MB -6.0%) vs baseline: +4.7% 🟡 recursivecomputation - 8/8✅ deepTime: ✅ 311.655ms (SLO: <320.950ms -2.9%) vs baseline: +0.3% Memory: ✅ 37.356MB (SLO: <38.750MB -3.6%) vs baseline: +4.1% ✅ deep-profiledTime: ✅ 328.531ms (SLO: <359.150ms -8.5%) vs baseline: -0.2% Memory: ✅ 43.785MB (SLO: <46.000MB -4.8%) vs baseline: +4.8% ✅ mediumTime: ✅ 7.296ms (SLO: <7.400ms 🟡 -1.4%) vs baseline: ~same Memory: ✅ 36.589MB (SLO: <38.000MB -3.7%) vs baseline: +4.7% ✅ shallowTime: ✅ 1.020ms (SLO: <1.050ms -2.8%) vs baseline: +1.5% Memory: ✅ 36.746MB (SLO: <38.000MB -3.3%) vs baseline: +4.6%
|
Codeowners resolved as |
…azel-pr4-payload-files
brettlangdon
left a comment
There was a problem hiding this comment.
docs and release note lgtm
There was a problem hiding this comment.
💡 Codex Review
dd-trace-py/ddtrace/testing/internal/session_manager.py
Lines 136 to 139 in b84ad23
Manifest mode forces itr_enabled/skipping_enabled off earlier in __init__, but the require_git refresh path fetches settings again and only reapplies env overrides. If cached settings have require_git=true, this can re-enable ITR/skipping after the manifest-mode override, violating the intended "always disabled in manifest mode" behavior and leaving session state inconsistent for later logic.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
## Description Adds Bazel-focused CI Visibility support with two offline execution modes, mirroring the Go implementation in DataDog/dd-trace-go#4503: - **Manifest mode** (`DD_TEST_OPTIMIZATION_MANIFEST_FILE`): reads settings, known tests, and test management data from pre-fetched JSON cache files inside `.testoptimization/`, enabling CI Visibility in Bazel's hermetic sandbox without network access. - **Payload-files mode** (`DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES`): writes test event, coverage, and telemetry payloads as JSON files to `TEST_UNDECLARED_OUTPUTS_DIR/payloads/{tests,coverage,telemetry}/` instead of sending HTTP requests. ### Key changes - **`offline_mode.py`**: `OfflineMode` singleton detects and validates both modes from env vars. Manifest version parsing supports plain `"1"` and `version=1` assignment syntax (matching Go). Runfiles resolution via `RUNFILES_DIR`, `RUNFILES_MANIFEST_FILE`, and `TEST_SRCDIR`. - **`cached_file_provider.py`**: `CachedFileDataProvider` implements the `TestOptDataProvider` protocol, reading from cache files. Skippable tests return empty unconditionally (hard no-op in manifest mode, matching Go). - **`writer.py`**: `TestOptWriter` and `TestCoverageWriter` intercept `_send_events` in payload-files mode to write JSON files. Filenames use `{kind}-{timestamp}-{pid}-{seq}.json` pattern matching Go's DDTestRunner expectations. Telemetry files use ordinal-first naming (`telemetry-{seq_padded}-{pid}.json`) for deterministic replay. - **`telemetry.py`**: `TelemetryAPI` accumulates CI Visibility metrics in payload-files mode and writes them to `payloads/telemetry/` on `finish()`, matching Go's telemetry file-sink behavior. - **`session_manager.py`**: Swaps `APIClient` for `CachedFileDataProvider` in manifest mode. Uses `NoOpBackendConnectorSetup` for writers/telemetry. Forces test skipping off in manifest mode. Skips git data upload in both offline modes. - **`env_tags.py`**: In payload-files mode, reads CI/Git tags from `DD_TEST_OPTIMIZATION_ENV_DATA_FILE` instead of invoking git CLI. Falls back to `ci.provider.name = "bazel"` when no other provider is detected. - **`constants.py`**: New `DD_TEST_OPTIMIZATION_ENV_DATA_FILE` constant. ## Testing - Unit tests added/updated across 4 test files covering all new functionality: - `test_offline_mode.py` — manifest version parsing (plain, assignment syntax, blank lines, invalid), runfiles resolution, `OfflineMode` initialization - `test_cached_file_provider.py` — cache reading, skippable tests hard no-op - `test_payload_files.py` — payload file naming (`{kind}-{ts}-{pid}-{seq}.json`), telemetry ordinal naming, telemetry file output on `TelemetryAPI.finish()` - `test_bazel_offline_session_manager.py` — provider selection, git upload skipping, env data file reading, bazel provider fallback, skipping forced off in manifest mode - All 90 tests pass on Python 3.12 with pytest ~=8.0. ## Risks - **Payload file naming**: Changed from `payload_<n>.json` to `{kind}-{ts}-{pid}-{seq}.json` to match Go's DDTestRunner expectations. Any consumer that relied on the old naming would need updating. - Manifest mode disables test skipping unconditionally — this is intentional and matches Go behavior. ## Additional Notes - Mirrors Go PR: DataDog/dd-trace-go#4503 - Features not ported (not applicable to Python's architecture): per-span CI/git tag stripping (Python sets tags at metadata level), impacted tests suppression (no such concept in Python yet), low-level git CLI guard (only called from guarded `upload_git_data`), CI log shipping suppression (Python plugin has no log shipping). Co-authored-by: federico.mon <federico.mon@datadoghq.com>
# What Does This Do Adds Bazel-focused CI Visibility support with two offline execution modes, mirroring [dd-trace-go#4503](DataDog/dd-trace-go#4503) and [dd-trace-py#17197](DataDog/dd-trace-py#17197): - **Manifest mode** (`DD_TEST_OPTIMIZATION_MANIFEST_FILE`): reads settings, known tests, flaky tests, and test management data from pre-fetched JSON cache files instead of hitting the backend. - **Payload-files mode** (`DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES`): writes CI test cycle, coverage, and tracer telemetry to `$TEST_UNDECLARED_OUTPUTS_DIR/payloads/{tests,coverage,telemetry}/*.json` instead of POSTing them. ## Key Changes - `BazelMode` (internal-api): detects both modes, resolves the manifest path via Bazel's rlocation algorithm, parses the `version=<int>` header, and exposes the `tests/`, `coverage/`, and `telemetry/` output directories. - `FileBasedConfigurationApi` (agent-ci-visibility): reads the same JSON envelopes as the HTTP API from disk; null paths return safe defaults. - `FileBasedPayloadDispatcher` (dd-trace-core): serializes CI test cycle and coverage spans as JSON files; strips `ci.*`/`git.*`/`runtime.*`/`os.*` tags to avoid cache invalidation; atomic temp-file + rename. Writes `trace_id`/`span_id`/`parent_id` as unsigned 64-bit JSON numbers (not strings) so backend schema validation passes. - `FileBasedTelemetryClient` (telemetry): subclass of `TelemetryClient` that writes the existing Moshi-encoded telemetry request body to a file; `TelemetryRouter` gets a single-client path that skips feature discovery; `TelemetrySystem` swaps in the file-based client when Bazel mode is active. - `WriterFactory` / `CiVisibilityServices` / `CiVisibilityRepoServices`: wire the file-based dispatcher/config API, disable the git client, and skip git-data upload when Bazel mode is active. - `CoreTracer`: in Bazel payload-files mode, uses `StreamingTraceCollector` (streams each CI Visibility span individually) and `DDIntakeTraceInterceptor` (not the APM-protocol interceptor, which strips `test_{session,module,suite}_end` spans) — same treatment as agentless, so all CITESTCYCLE events reach the file dispatcher. - `JUnit4TracingListener` / `JUnit4Utils`: lazy-register the test suite in `testStarted` so runners that don't fire `testSuiteStarted` still produce a proper suite span; unwrap `com.google.testing.junit.junit4.runner.RunNotifierWrapper` in `runListenersFromRunNotifier` so the idempotency check sees listeners installed on the inner notifier (fixes duplicate-listener installation under `BazelTestRunner`). - `Config`: adds `DD_TEST_OPTIMIZATION_MANIFEST_FILE` and `DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES`; skips API-key validation in these modes. `TEST_UNDECLARED_OUTPUTS_DIR` is read directly in `BazelMode` (it's a Bazel-provided env var, not a DD config). # Motivation Bazel can run tests in hermetic sandboxes with no network access. The existing CI Visibility pipeline requires HTTP calls to fetch configuration and submit payloads, which is incompatible with Bazel's execution model. Most of our operations, such as tagging tests with git metadata, also invalid Bazel's cache. This PR enables CI Visibility under Bazel by reading configuration from pre-fetched cache files and writing payloads/telemetry to files, with the orchestration of everything else being handled by our custom testing rule. # Additional Notes - Unit tests cover each new component: `BazelModeTest`, `FileBasedConfigurationApiTest` (shares the existing `*-response.ftl` fixtures with `ConfigurationApiImplTest` to keep the HTTP and file code paths in sync), `FileBasedPayloadDispatcherTest`, `FileBasedTelemetryClientTest`, and extended `TelemetryRouterSpecification`. - End-to-end repro validated locally against `DataDog/rules_test_optimization_tests`: 3 `test` + 1 `test_suite_end` + 1 `test_module_end` + 1 `test_session_end` events emitted to the payload file, no duplicate listener errors, no schema-validation failures. - Future work, not included in this PR to avoid changes too big: - Include instrumentation improvements to better handle Bazel's custom JUnit4 test runner - Refactoring of configuration API related DTOs to common utilities - Define a specification for mapping and serializing CI Vis spans to avoid logic mirroring between the two approaches (original vs file-based) - Possibly introduce a smoke test for e2e testing of bazel process, to avoid dependencies on an external repository. # Contributor Checklist - [x] Format the title according to [the contribution guidelines](https://github.com/DataDog/dd-trace-java/blob/master/CONTRIBUTING.md#title-format) - [ ] Assign the `type:` and (`comp:` or `inst:`) labels in addition to [any other useful labels](https://github.com/DataDog/dd-trace-java/blob/master/CONTRIBUTING.md#labels) - [ ] Update the [CODEOWNERS](https://github.com/DataDog/dd-trace-java/blob/master/.github/CODEOWNERS) file on source file addition, migration, or deletion - [ ] Update [public documentation](https://docs.datadoghq.com/tracing/trace_collection/library_config/java/) with any new configuration flags or behaviors Jira ticket: [SDTEST-3335] ***Note:*** **Once your PR is ready to merge, add it to the merge queue by commenting \`/merge\`.** \`/merge -c\` cancels the queue request. \`/merge -f --reason "reason"\` skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see [this doc](https://datadoghq.atlassian.net/wiki/spaces/DEVX/pages/3121612126/MergeQueue). [SDTEST-3335]: https://datadoghq.atlassian.net/browse/SDTEST-3335?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: daniel.mohedano <daniel.mohedano@datadoghq.com>
## Description Adds Bazel-focused CI Visibility support with two offline execution modes, mirroring the Go implementation in DataDog/dd-trace-go#4503: - **Manifest mode** (`DD_TEST_OPTIMIZATION_MANIFEST_FILE`): reads settings, known tests, and test management data from pre-fetched JSON cache files inside `.testoptimization/`, enabling CI Visibility in Bazel's hermetic sandbox without network access. - **Payload-files mode** (`DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES`): writes test event, coverage, and telemetry payloads as JSON files to `TEST_UNDECLARED_OUTPUTS_DIR/payloads/{tests,coverage,telemetry}/` instead of sending HTTP requests. ### Key changes - **`offline_mode.py`**: `OfflineMode` singleton detects and validates both modes from env vars. Manifest version parsing supports plain `"1"` and `version=1` assignment syntax (matching Go). Runfiles resolution via `RUNFILES_DIR`, `RUNFILES_MANIFEST_FILE`, and `TEST_SRCDIR`. - **`cached_file_provider.py`**: `CachedFileDataProvider` implements the `TestOptDataProvider` protocol, reading from cache files. Skippable tests return empty unconditionally (hard no-op in manifest mode, matching Go). - **`writer.py`**: `TestOptWriter` and `TestCoverageWriter` intercept `_send_events` in payload-files mode to write JSON files. Filenames use `{kind}-{timestamp}-{pid}-{seq}.json` pattern matching Go's DDTestRunner expectations. Telemetry files use ordinal-first naming (`telemetry-{seq_padded}-{pid}.json`) for deterministic replay. - **`telemetry.py`**: `TelemetryAPI` accumulates CI Visibility metrics in payload-files mode and writes them to `payloads/telemetry/` on `finish()`, matching Go's telemetry file-sink behavior. - **`session_manager.py`**: Swaps `APIClient` for `CachedFileDataProvider` in manifest mode. Uses `NoOpBackendConnectorSetup` for writers/telemetry. Forces test skipping off in manifest mode. Skips git data upload in both offline modes. - **`env_tags.py`**: In payload-files mode, reads CI/Git tags from `DD_TEST_OPTIMIZATION_ENV_DATA_FILE` instead of invoking git CLI. Falls back to `ci.provider.name = "bazel"` when no other provider is detected. - **`constants.py`**: New `DD_TEST_OPTIMIZATION_ENV_DATA_FILE` constant. ## Testing - Unit tests added/updated across 4 test files covering all new functionality: - `test_offline_mode.py` — manifest version parsing (plain, assignment syntax, blank lines, invalid), runfiles resolution, `OfflineMode` initialization - `test_cached_file_provider.py` — cache reading, skippable tests hard no-op - `test_payload_files.py` — payload file naming (`{kind}-{ts}-{pid}-{seq}.json`), telemetry ordinal naming, telemetry file output on `TelemetryAPI.finish()` - `test_bazel_offline_session_manager.py` — provider selection, git upload skipping, env data file reading, bazel provider fallback, skipping forced off in manifest mode - All 90 tests pass on Python 3.12 with pytest ~=8.0. ## Risks - **Payload file naming**: Changed from `payload_<n>.json` to `{kind}-{ts}-{pid}-{seq}.json` to match Go's DDTestRunner expectations. Any consumer that relied on the old naming would need updating. - Manifest mode disables test skipping unconditionally — this is intentional and matches Go behavior. ## Additional Notes - Mirrors Go PR: DataDog/dd-trace-go#4503 - Features not ported (not applicable to Python's architecture): per-span CI/git tag stripping (Python sets tags at metadata level), impacted tests suppression (no such concept in Python yet), low-level git CLI guard (only called from guarded `upload_git_data`), CI log shipping suppression (Python plugin has no log shipping). Co-authored-by: federico.mon <federico.mon@datadoghq.com>
Description
Adds Bazel-focused CI Visibility support with two offline execution modes, mirroring the Go implementation in DataDog/dd-trace-go#4503:
DD_TEST_OPTIMIZATION_MANIFEST_FILE): reads settings, known tests, and test management data from pre-fetched JSON cache files inside.testoptimization/, enabling CI Visibility in Bazel's hermetic sandbox without network access.DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES): writes test event, coverage, and telemetry payloads as JSON files toTEST_UNDECLARED_OUTPUTS_DIR/payloads/{tests,coverage,telemetry}/instead of sending HTTP requests.Key changes
offline_mode.py:OfflineModesingleton detects and validates both modes from env vars. Manifest version parsing supports plain"1"andversion=1assignment syntax (matching Go). Runfiles resolution viaRUNFILES_DIR,RUNFILES_MANIFEST_FILE, andTEST_SRCDIR.cached_file_provider.py:CachedFileDataProviderimplements theTestOptDataProviderprotocol, reading from cache files. Skippable tests return empty unconditionally (hard no-op in manifest mode, matching Go).writer.py:TestOptWriterandTestCoverageWriterintercept_send_eventsin payload-files mode to write JSON files. Filenames use{kind}-{timestamp}-{pid}-{seq}.jsonpattern matching Go's DDTestRunner expectations. Telemetry files use ordinal-first naming (telemetry-{seq_padded}-{pid}.json) for deterministic replay.telemetry.py:TelemetryAPIaccumulates CI Visibility metrics in payload-files mode and writes them topayloads/telemetry/onfinish(), matching Go's telemetry file-sink behavior.session_manager.py: SwapsAPIClientforCachedFileDataProviderin manifest mode. UsesNoOpBackendConnectorSetupfor writers/telemetry. Forces test skipping off in manifest mode. Skips git data upload in both offline modes.env_tags.py: In payload-files mode, reads CI/Git tags fromDD_TEST_OPTIMIZATION_ENV_DATA_FILEinstead of invoking git CLI. Falls back toci.provider.name = "bazel"when no other provider is detected.constants.py: NewDD_TEST_OPTIMIZATION_ENV_DATA_FILEconstant.Testing
test_offline_mode.py— manifest version parsing (plain, assignment syntax, blank lines, invalid), runfiles resolution,OfflineModeinitializationtest_cached_file_provider.py— cache reading, skippable tests hard no-optest_payload_files.py— payload file naming ({kind}-{ts}-{pid}-{seq}.json), telemetry ordinal naming, telemetry file output onTelemetryAPI.finish()test_bazel_offline_session_manager.py— provider selection, git upload skipping, env data file reading, bazel provider fallback, skipping forced off in manifest modeRisks
payload_<n>.jsonto{kind}-{ts}-{pid}-{seq}.jsonto match Go's DDTestRunner expectations. Any consumer that relied on the old naming would need updating.Additional Notes
upload_git_data), CI log shipping suppression (Python plugin has no log shipping).