Skip to content

Add a fetch-compute-concurrency option for toolkit#675

Merged
chrispalaskas merged 7 commits into
mainfrom
performance/add-fetch-compute-concurrency
Feb 23, 2026
Merged

Add a fetch-compute-concurrency option for toolkit#675
chrispalaskas merged 7 commits into
mainfrom
performance/add-fetch-compute-concurrency

Conversation

@chrispalaskas

Copy link
Copy Markdown
Contributor

Overview

I added an argument for the MN Toolkit which will limit the number of compute tokio tasks. The reason we need that is that we are already running the toolkit multithreaded, and each thread has to update the toolkit.db (which is another area of improvement). That leads to massive overburdening of the system.

🗹 TODO before merging

  • Ready

📌 Submission Checklist

  • Changes are backward-compatible (or flagged if breaking)
  • Pull request description explains why the change is needed
  • Self-reviewed the diff
  • I have included a change file, or skipped for this reason:
  • If the changes introduce a new feature, I have bumped the node minor version
  • Update documentation (if relevant)
  • Updated AGENTS.md if build commands, architecture, or workflows changed
  • No new todos introduced

🧪 Testing Evidence

Tested locally, indeed it fetches extremely slow with 1 compute worker.

Please describe any additional testing aside from CI:

  • Additional tests are provided (if possible)

🔱 Fork Strategy

  • Node Runtime Update
  • Node Client Update
  • Other:
  • N/A

Links

@chrispalaskas chrispalaskas requested a review from a team as a code owner February 13, 2026 15:43
@chrispalaskas chrispalaskas requested a review from ozgb February 13, 2026 15:43
@github-actions

github-actions Bot commented Feb 13, 2026

Copy link
Copy Markdown
Contributor

kics-logo

KICS version: v2.1.16

Category Results
CRITICAL CRITICAL 0
HIGH HIGH 0
MEDIUM MEDIUM 96
LOW LOW 12
INFO INFO 83
TRACE TRACE 0
TOTAL TOTAL 191
Metric Values
Files scanned placeholder 31
Files parsed placeholder 31
Files failed to scan placeholder 0
Total executed queries placeholder 73
Queries failed to execute placeholder 0
Execution time placeholder 9

@ozgb

ozgb commented Feb 16, 2026

Copy link
Copy Markdown
Contributor

Thanks for the PR - please add a change file under changes/added/ with a name describing the change (e.g. toolkit-add-fetch-compute-concurrency.md) in the following format:

#toolkit
# Add a fetch-compute-concurrency option for toolkit

<description>

PR: <pr link>
Ticket: https://shielded.atlassian.net/browse/PM-21786

@chrispalaskas

Copy link
Copy Markdown
Contributor Author

@ozgb change file added

@chrispalaskas chrispalaskas added this pull request to the merge queue Feb 23, 2026
Merged via the queue into main with commit ba4e67c Feb 23, 2026
38 checks passed
@chrispalaskas chrispalaskas deleted the performance/add-fetch-compute-concurrency branch February 23, 2026 10:51
gilescope pushed a commit that referenced this pull request Apr 8, 2026
m2ux added a commit that referenced this pull request Apr 23, 2026
Signed-off-by: Mike Clay <mike.clay@shielded.io>
m2ux added a commit that referenced this pull request Apr 23, 2026
Signed-off-by: Mike Clay <mike.clay@shielded.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants