Adding benchmarking tool to compare performance between PRs and releases#868
Adding benchmarking tool to compare performance between PRs and releases#868jdmarshall merged 4 commits intonode-config:masterfrom
Conversation
📝 WalkthroughWalkthroughAdded a benchmark suite and configs; CI now runs Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant CLI as "CLI: npm run benchmarks"
participant Runner as "benchmarks/index.js"
participant Loader as "Config Loader/Util"
participant Faceoff as "faceoff"
participant Process as "process/console"
CLI->>Runner: start Node with flags
Runner->>Loader: set NODE_ENV & load SAMPLE_DATA (benchmarks/config/*)
Runner->>Faceoff: register benchmark suites (access, util, config.util)
Faceoff->>Runner: execute suites and collect metrics
Runner->>Process: log results
alt failures present
Runner->>Process: exit non-zero
else
Runner->>Process: exit 0
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@benchmarks/index.js`:
- Around line 24-25: The object key 'branch ' contains an unintended trailing
space; update the key to 'branch' (remove the trailing space) in the mapping
where it's defined (the entry currently written as 'branch ') so downstream
lookups and benchmark output use the correct key name; check any code that
references 'branch ' and update those references to 'branch' if present.
- Around line 6-9: Move the environment setup so NODE_ENV and NODE_CONFIG_DIR
are assigned before invoking loadConfig; specifically, set process.env.NODE_ENV
= 'benchmark' and process.env.NODE_CONFIG_DIR = configDir prior to the call to
loadConfig() that initializes SAMPLE_DATA, ensuring the SAMPLE_DATA =
loadConfig() call reads the correct environment-dependent config.
🧹 Nitpick comments (4)
.github/workflows/tests.yml (1)
22-23: Consider upgradingactions/setup-nodeto v4.
actions/checkout@v4is used on line 19, butactions/setup-node@v2is outdated. For consistency and to benefit from bug fixes and performance improvements, consider upgrading to v4.Suggested fix
- name: Setup Node.js - uses: actions/setup-node@v2 + uses: actions/setup-node@v4 with: node-version: '22'package.json (1)
60-62: Consider renamingbuildscript to clarify its purpose.The
buildscript name typically implies compilation/transpilation, but here it runs tests and benchmarks. This could be confusing for contributors. Consider a more descriptive name likeciorcheck.That said, this is a minor naming concern and the current approach is functional.
benchmarks/config/default.mjs (1)
1-9: Minor: Inconsistent property quoting style.
EnvOverrideuses unquoted keys whileAnotherModuleuses quoted keys. Consider using a consistent style throughout for readability.Suggested fix
export default { EnvOverride: { parm_number_1: "from_default_mjs", parm2: 22 }, - "AnotherModule": { - "parm10": "value10" + AnotherModule: { + parm10: "value10" } };benchmarks/index.js (1)
29-38: Unused variablesvalmay cause linter warnings.The
valvariables are assigned but never read, which could trigger ESLintno-unused-varswarnings. While this is intentional for benchmarking purposes, consider using a void expression or disabling the lint rule locally.Suggested fix to suppress warnings
suite.add('get', (config, keys) => { for (const key of keys) { - let val = config[key]; + void config[key]; } }); suite.add('get()', (config, keys) => { for (const key of keys) { - let val = config.get(key); + void config.get(key); } });
released versions of node-config.
a5f858b to
9f803ad
Compare
Tweak tests to reduce inconclusive outcomes.
This is a tool I wrote while working on prom-client, which used a similar tool that was defunct.
What I found is that it gets confusing looking at multiple PRs and seeing if anyone is being blamed for regressions or double counting improvements that have landed in trunk from another PR. So I extended the functionality to allow git urls as version numbers. This wraps a library called bench-node, which is authored by a member of the nodejs team.
The set of versions being checked here is excessive, but I wanted to verify that none of the code I had written or merged since becoming a maintainer had introduced any problems, and I'm not seeing them, at least in the paths currently being exercised here. We can delete a few of these as soon as the benchmark count gets larger but we're setting at 20 seconds for these tests.
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.