Replace d3-queue with native async/await across test infrastructure by DennisOSRM · Pull Request #7458 · Project-OSRM/osrm-backend

DennisOSRM · 2026-04-08T20:58:11Z

Removes d3-queue dependency and replaces all usages with native async/await and Promise-based patterns. Follow-up to review feedback requiring fully promise-based APIs with no callback mixing.

Core changes

features/support/data.js: processRowsAndDiff, reprocessAndLoadData, and buildWaysFromTable drop their callback parameters entirely — errors throw, callers await
features/step_definitions/routability.js: testRoutabilityRow returns a Promise instead of calling cb; testRow is now async; step function is async
All call sites (matching.js, nearest.js, trip.js, distance_matrix.js, shared_steps.js, routing.js, requests.js): step functions converted to async, inner testRow/requestRow functions wrap callback-based HTTP APIs in new Promise() and return results directly

Before / After

// Before: mixed async+callback
async processRowsAndDiff(table, fn, callback) {
  try { /* ... */ callback(); }
  catch (err) { callback(err); }
}
this.processRowsAndDiff(table, testRow, callback); // caller ignores returned Promise

// After: pure async
async processRowsAndDiff(table, fn) {
  for (const [i, row] of table.hashes().entries()) {
    actual.push(await fn(row, i)); // fn returns a Promise
  }
  if (diff) throw new Error(diff);
}
await this.processRowsAndDiff(table, testRow);

Remove the stale d3-queue dependency and replace all usages with native async/await and Promise.all across the test infrastructure: - features/lib/hash.js: convert hashOfFiles/hashOfFile to async, use Promise.all + fs.promises.readFile - features/support/data.js: convert processRowsAndDiff to async sequential loop; simplify buildWaysFromTable to plain forEach - features/step_definitions/data.js: remove queue wrappers from synchronous steps (shortcuts, ways, relations) - features/step_definitions/routing.js: replace queue(1) serial loop with async for-loop - features/step_definitions/routability.js: replace queue(1) forw/backw sequential run and synchronous parseRes queue with async/await Closes #7453

Copilot

Pull request overview

This PR removes the d3-queue dependency from the Cucumber test infrastructure and rewrites affected helpers/step-definitions to use native Promises (async/await, Promise.all) instead.

Changes:

Drop d3-queue from npm devDependencies and refactor usages across feature support + step definitions.
Convert serial queue(1) patterns to explicit await loops for deterministic sequencing.
Switch file hashing helpers to fs.promises.readFile and Promise-based APIs.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
package.json	Removes `d3-queue`; adds an eslint-specific AJV override.
package-lock.json	Removes `d3-queue` and updates lockfile resolution/metadata (incl. engines + transitive updates).
features/support/data.js	Removes `d3-queue`; simplifies synchronous table-to-OSM building; rewrites row processing to sequential `await`.
features/lib/hash.js	Refactors hashing helpers to async Promise-based functions.
features/step_definitions/data.js	Removes `d3-queue` wrappers from synchronous setup steps (shortcuts/ways/relations).
features/step_definitions/routing.js	Replaces serial `queue(1)` routing loop with an `await`-driven loop.
features/step_definitions/routability.js	Replaces sequential `queue(1)` direction tests and result comparison with `async`/`await`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

DennisOSRM · 2026-04-09T07:45:39Z

@copilot apply changes based on the comments in this thread. Make it async/await all the way and remove the callback. The functions should be a promise and callsites should await

… pure async/await, remove all callbacks Agent-Logs-Url: https://github.com/Project-OSRM/osrm-backend/sessions/37a703d6-0f13-4686-8bd1-53c2cef81981 Co-authored-by: DennisOSRM <1067895+DennisOSRM@users.noreply.github.com>

Copilot · 2026-04-09T08:00:37Z

@copilot apply changes based on the comments in this thread. Make it async/await all the way and remove the callb...

Done in commit 03a5db9. Converted both processRowsAndDiff and reprocessAndLoadData in data.js to pure async functions with no callback parameters (errors now throw, callers await). Updated testRoutabilityRow to return the result instead of calling cb. All call sites across routability.js, matching.js, nearest.js, trip.js, distance_matrix.js, shared_steps.js (via WhenIRouteIShouldGet), routing.js, and requests.js now use async function + await throughout.

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (2)

features/step_definitions/trip.js:29

afterRequest writes into params[...], but params is only declared inside the non-request branch later in the function, so it is out of scope here and will throw a ReferenceError as soon as a param:* column is present. Declare params in the outer testRow scope (e.g., let params = this.queryParams) and ensure both branches use that same variable (or pass it into afterRequest).

        for (const k in row) {
          const match = k.match(/param:(.*)/);
          if (match) {
            if (row[k] === '(nil)') {
              params[match[1]] = null;
            } else if (row[k]) {
              params[match[1]] = [row[k]];
            }
            got[k] = row[k];
          }

features/step_definitions/trip.js:135

In the row.request branch, got is used before being initialized (got.request = ...), which will throw a TypeError. Initialize got (e.g., got = { request: row.request }) before assigning fields in this branch.

      if (row.request) {
        got.request = row.request;
        this.requestUrl(row.request, afterRequest);
      } else {

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-09T08:59:34Z

              if (!row.time.match(/\d+s/))
-                return cb(
+                return reject(
                  new Error('*** Time must be specied in seconds. (ex: 60s)'),
                );


Typo in error message: "specied" should be "specified" (this message is user-facing when tests fail).

Copilot · 2026-04-09T08:59:34Z

                if (!row.speed.match(/\d+ km\/h/))
-                  cb(
+                  return reject(
                    new Error(
                      '*** Speed must be specied in km/h. (ex: 50 km/h)',
                    ),


Typo in error message: "specied" should be "specified" (this message is user-facing when tests fail).

Copilot · 2026-04-09T08:59:35Z

              if (k.match(/^a:/)) {
                const a_type = k.slice(2);
                if (whitelist.indexOf(a_type) == -1)
-                  return cb(new Error('Unrecognized annotation field', a_type));
+                  return reject(new Error('Unrecognized annotation field', a_type));
                if (annotation && !annotation[a_type])
-                  return cb(
+                  return reject(
                    new Error('Annotation not found in response', a_type),
                  );


new Error('Unrecognized annotation field', a_type) ignores the second argument, so the annotation type is lost from the error message. Include a_type in the message (e.g., via template string / util.format) so failures are actionable.

Copilot · 2026-04-09T08:59:35Z

              } else if (k.match(/^am:/)) {
                const a_type = k.slice(3);
                if (metadata_whitelist.indexOf(a_type) == -1)
-                  return cb(new Error('Unrecognized annotation field', a_type));
+                  return reject(new Error('Unrecognized annotation field', a_type));
                if (
                  annotation &&
                  (!annotation.metadata || !annotation.metadata[a_type])
                )
-                  return cb(
+                  return reject(
                    new Error('Annotation not found in response', a_type),
                  );


Same issue here: new Error('Unrecognized annotation field', a_type) / new Error('Annotation not found in response', a_type) drop a_type because Error doesn't accept a second message argument. Format the message to include the field name.

Copilot · 2026-04-09T08:59:35Z

+
+  if (!testedHeaders.some((k) => !!headers.has(k))) {
+    throw new Error(
+      '*** routability table must contain either "forw", "backw", "bothw", "forw_rate" or "backw_mode" column',


The validation error mentions a "backw_mode" column, but this step actually checks for backw / backw_rate / bothw_rate (see testedHeaders). Update the message so it reflects the accepted columns.

Suggested change

'*** routability table must contain either "forw", "backw", "bothw", "forw_rate" or "backw_mode" column',

'*** routability table must contain at least one of the following columns: "forw", "backw", "bothw", "forw_rate", "backw_rate", or "bothw_rate"',

- Fix 'specied' typos in time and speed error messages - Include annotation field name in Unrecognized/NotFound error messages - Fix routability error message to list correct column names - Rename unused 'ri' param to '_ri' to satisfy ESLint Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

DennisOSRM marked this pull request as ready for review April 8, 2026 21:10

Copilot AI review requested due to automatic review settings April 8, 2026 21:10

Copilot started reviewing on behalf of DennisOSRM April 8, 2026 21:14 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Comment thread features/lib/hash.js Outdated

Comment thread features/step_definitions/routability.js

DennisOSRM and others added 2 commits April 9, 2026 07:36

Address reviewer feedback

b55d7af

Merge branch 'master' into dennis/replace_d3_queue

406cf58

DennisOSRM requested a review from Copilot April 9, 2026 07:14

Copilot started reviewing on behalf of DennisOSRM April 9, 2026 07:15 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Comment thread features/support/data.js Outdated

Comment thread features/step_definitions/routability.js Outdated

Copilot started work on behalf of DennisOSRM April 9, 2026 07:45 View session

Copilot AI changed the title ~~Replace d3-queue with native Promise.all and async/await~~ Replace d3-queue with native async/await across test infrastructure Apr 9, 2026

Copilot finished work on behalf of DennisOSRM April 9, 2026 08:02

DennisOSRM requested a review from Copilot April 9, 2026 08:54

Copilot started reviewing on behalf of DennisOSRM April 9, 2026 08:55 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

DennisOSRM and others added 2 commits April 10, 2026 20:27

Merge branch 'master' into dennis/replace_d3_queue

92f6505

DennisOSRM merged commit 24d5324 into master Apr 10, 2026
23 checks passed

DennisOSRM deleted the dennis/replace_d3_queue branch April 10, 2026 20:07

	'*** routability table must contain either "forw", "backw", "bothw", "forw_rate" or "backw_mode" column',
	'*** routability table must contain at least one of the following columns: "forw", "backw", "bothw", "forw_rate", "backw_rate", or "bothw_rate"',

Uh oh!

Conversation

DennisOSRM commented Apr 8, 2026 • edited by Copilot AI Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Core changes

Before / After

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

DennisOSRM commented Apr 9, 2026

Uh oh!

Copilot AI commented Apr 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

DennisOSRM commented Apr 8, 2026 •

edited by Copilot AI

Loading