Conversation
… batches - Replace complex OR queries with efficient IN clauses in updateManySecret, createManySecret, and deleteManySecret - Batch bulkUpdate DAL method to process updates in chunks of 500 - Batch insertMany calls for secret versions and secret inserts in fnSecretBulkInsert and fnSecretBulkUpdate The complex OR queries generated N individual AND conditions (one per secret) which produced enormous SQL queries for large batches. Since all conditions share type=Shared, a single WHERE type='shared' AND key IN (...) is equivalent and far more efficient. The bulkUpdate method previously fired all N individual UPDATE queries via Promise.all simultaneously. Now processes in batches of 500 to prevent transaction timeouts. Similarly, insertMany calls for secret versions and secrets are now batched to avoid potential PostgreSQL parameter limits on very large inserts. Co-Authored-By: unknown <>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Greptile SummaryThis PR optimizes bulk secret operations to prevent 502/500 timeouts when upserting large batches (740+ secrets). The changes address two key inefficiencies:
The query optimization is semantically equivalent and proven safe. Batching preserves array order correctly, maintaining index alignment for metadata and parent version IDs. Confidence Score: 4/5
Important Files Changed
Last reviewed commit: 4669eed |
Additional Comments (1)
Security Impact: High - Could lead to unauthorized data deletion Fix: Should also be converted to use Context Used: Rule from ObjectiveFlag database queries t... (source) |
fix: resolve eslint no-await-in-loop and prettier formatting issues fix: add type annotations and extract shared BATCH_SIZE constant per review feedback fix: correct import sort order for TSecretVersionsV2 Co-Authored-By: unknown <>
ae233b5 to
4669eed
Compare
|
@devin I'd like to hear your thoughts on this
|
|
@varonix0 Good question — I looked into this carefully. The Greptile "critical security vulnerability" finding is a false positive. The .where({ folderId }) // outer AND condition
.where((bd) => { // grouped inner condition (wrapped in parentheses)
data.forEach((el) => {
void bd.orWhere({ ... }); // OR conditions scoped inside the group
});
})This generates SQL like: DELETE FROM secrets_v2
WHERE folder_id = ?
AND ((key = ? AND type = ?) OR (key = ? AND type = ?) OR ...)The Additionally, Greptile's suggested
That said, two valid points:
If you'd like, I can optimize |
|
@greptile re-review this PR with previous comments and addressed feedback in mind |
Context
Customers report 502 errors when bulk upserting ~740 secrets via
PATCH /api/v3/secrets/batch/rawwithmode: "upsert", and 500 errors even when chunking down to 50 secrets. This blocks migrations.Root cause: Two compounding inefficiencies in the bulk secret write path:
Massive OR queries:
updateManySecret,createManySecret, anddeleteManySecretconstructed a$complexOR clause with N individual(key = ? AND type = ?)conditions. Since every condition usestype = 'shared', this is equivalent to a singleWHERE type = 'shared' AND key IN (...), which is dramatically more efficient for the query planner.Unbatched writes:
bulkUpdatefired all N individual UPDATE queries viaPromise.allsimultaneously, andinsertManyfor secrets/versions sent all rows in a single query—both problematic at scale.Changes:
$complexOR queries with$in+ directtypefilter (4 call sites insecret-v2-bridge-service.ts)bulkUpdatein the DAL to process 500 updates per iteration (secret-v2-bridge-dal.ts)insertManycalls for secrets and secret versions infnSecretBulkInsertandfnSecretBulkUpdate(secret-v2-bridge-fns.ts)Important notes for reviewers
$inpattern ([${TableName.SecretV2}.key as "key"]) is already used elsewhere in the same files (e.g.,fnSecretBulkUpdatefinalfindcall), so it's a proven pattern.BULK_BATCH_SIZE = 500is extracted as a shared constant insecret-v2-bridge-fns.ts(used in 3 places). The DAL still has its own localBATCH_SIZE = 500—could be unified if preferred.bulkUpdatestill usesPromise.all(i.e., 500 concurrent UPDATE queries per batch). This is much better than 740 at once but still parallel within the batch—worth confirming this is acceptable for your DB connection pool.Human review checklist
$inqueries behave identically to the old$complexOR queries for all edge cases (e.g., empty arrays, duplicate keys)$inpattern with the computed column name[${TableName.SecretV2}.key as "key"]produces correct SQL in all four call sitesinsertManyresults maintain correct index alignment withinputSecrets(used for tag mapping and version metadata)bulkUpdateis safe for your production DB connection poolSteps to verify the change
INclauses instead of massive OR chainsUpdates since last revision
newSecrets: TSecretsV2[]andsecretVersions: TSecretVersionsV2[]for all batched arrays (addresses Greptile review feedback)BULK_BATCH_SIZE = 500as a shared constant insecret-v2-bridge-fns.tsinstead of hardcoding per loopTSecretsV2beforeTSecretVersionsV2) to satisfysimple-import-sort/imports{ tx: undefined }parameter fromfind()call increateManySecret(addresses Greptile review feedback)Type
Checklist
type(scope): short description(scope is optional, e.g.,fix: prevent crash on syncorfix(api): handle null response).Link to Devin run: https://app.devin.ai/sessions/006c60068ec248608f44d60ce8ca7792
Requested by: @0xArshdeep