Skip to content

Commit 8cca78d

Browse files
TheLastCicadaTheLastCicada
committed
feat(APP): purge removed-org data on orglist reconcile
When ONLY_CADT_SUBSCRIPTIONS removes an organization that is no longer on the governance orgList, also delete the organization record and all of the registry data it created, instead of only unsubscribing and retaining the data. Deletion is unconditional and does not check whether other orgs reference the data. - add purgeV2OrganizationData to hard-delete every V2 row owned by an org (org_uid tables plus project/unit/verification/aef-t1 traced children) and wire it into OrganizationsV2.deleteAllOrganizationData, which previously left projects/units/etc orphaned - rename the reconcile helper to removeOrgsNotInOrgList: unsubscribe (when still subscribed) then delete; skip home org, governance body, and PENDING; purge orgs already locally unsubscribed but off the orgList - add skipStagingTruncate so background removal of a remote org cannot wipe the home org's pending staged changes - add recordUserDeleted (default true); background reconcile passes false so an off-orglist removal does not populate the user-deleted suppression list - release add/delete + sync mutexes via an idempotent helper on every exit path (incl. post-commit meta write and a throwing rollback) and open the transaction inside the try to avoid leaking the locks - log purged row counts and a per-cycle summary with a warn tripwire on mass removals or failures
1 parent 3a4e6c7 commit 8cca78d

10 files changed

Lines changed: 1131 additions & 168 deletions

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -277,7 +277,7 @@ In the `CHIA_ROOT` directory (usually `~/.chia/mainnet` on Linux), CADT will add
277277
* **DATALAYER_FILE_SERVER_URL**: Publicly available URL and port where Chia Datalayer [files are served](#datalayer-http-file-serving), including schema (http:// or https://). If serving DataLayer files from S3, this would be the public URL of the S3 bucket. Port can be omitted if using standard ports for http or https requests.
278278
* **AUTO_SUBSCRIBE_FILESTORE**: Subscribing to the filestore for any organization is optional. To automatically subscribe and sync the filestore to every organization you subscribe to, set this to `true`.
279279
* **AUTO_MIRROR_EXTERNAL_STORES**: When set to true (the default), CADT will automatically create mirrors for each store you are subscribed to. Mirroring all subscriptions using the `DATALAYER_FILE_SERVER_URL` will make the entire CADT network more resilient and distributed. Note: `DATALAYER_FILE_SERVER_URL` must also be set to a valid URL or IP address for mirrors to be created. Both settings are required for external store mirroring to function.
280-
* **ONLY_CADT_SUBSCRIPTIONS**: When `true` (the default), CADT keeps DataLayer subscriptions aligned with the governance **orgList** in both directions. Organizations removed from the orgList are unsubscribed from DataLayer (`subscribed: false`); already-synced registry data on this node is **not** deleted. Organizations on the orgList that are not subscribed are subscribed (including orgs re-added after a prior removal, including orgs previously removed via the API delete flow). The home organization and governance body store are never auto-unsubscribed. Reconciliation runs only when a **non-empty** orgList is present locally (after a successful governance sync); an empty orgList is treated as “not ready” and does not trigger unsubscribes. Set to `false` to disable orglist-driven subscribe/unsubscribe reconciliation. While enabled, a manual unsubscribe of an org still listed on the orgList will be reverted on the next sync cycle.
280+
* **ONLY_CADT_SUBSCRIPTIONS**: When `true` (the default), CADT keeps DataLayer subscriptions aligned with the governance **orgList** in both directions. Organizations removed from the orgList are unsubscribed from DataLayer **and completely removed from this node** — the organization record and **all** of its registry data (projects, units, and every related record it created) are deleted from the database. The deletion is unconditional: it does **not** check whether another organization references that data. Organizations on the orgList that are not subscribed are subscribed (including orgs re-added after a prior removal, including orgs previously removed via the API delete flow). The home organization and governance body store are never auto-removed. Reconciliation runs only when a **non-empty** orgList is present locally (after a successful governance sync); an empty orgList is treated as “not ready” and does not trigger removals. Set to `false` to disable orglist-driven subscribe/remove reconciliation. While enabled, a manual unsubscribe of an org still listed on the orgList will be reverted on the next sync cycle.
281281
* **LOG_LEVEL**: Controls verbosity of logging. Common settings are `info` and `debug`. Setting to `silly` will log all queries.
282282
* **TASKS**: Section for configuring sync intervals.
283283
* **GOVERNANCE_SYNC_TASK_INTERVAL**: Syncs picklist, orgList, and glossary from the governance node. Default 30 seconds.

‎src/models/organizations/organizations.model.js‎

Lines changed: 57 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1641,8 +1641,21 @@ class Organization extends Model {
16411641
/**
16421642
* removes all records of an organization from all models with an `orgUid` column
16431643
* @param orgUid
1644+
* @param {object} [options]
1645+
* @param {boolean} [options.skipStagingTruncate=false] - When true, the global
1646+
* staging table is left untouched. Staging holds only the home org's pending
1647+
* (uncommitted) changes and has no orgUid column, so background callers that
1648+
* remove a remote org (e.g. orglist subscription reconcile) must pass `true`
1649+
* to avoid discarding the operator's own staged work.
1650+
* @param {boolean} [options.recordUserDeleted=true] - When true, the org is
1651+
* recorded in the meta user-deleted list so default-org sync will not
1652+
* re-import it. Background orglist reconcile passes `false`: an org removed
1653+
* because it left the governance orgList is not a user deletion.
16441654
*/
1645-
static async deleteAllOrganizationData(orgUid) {
1655+
static async deleteAllOrganizationData(
1656+
orgUid,
1657+
{ skipStagingTruncate = false, recordUserDeleted = true } = {},
1658+
) {
16461659
logger.verbose('[v1]: acquiring add/delete org mutex to delete organization');
16471660
const releaseAddDeleteMutex =
16481661
await addOrDeleteOrganizationRecordMutex.acquire();
@@ -1653,32 +1666,68 @@ class Organization extends Model {
16531666
const releaseAuditTransactionMutex =
16541667
await processingSyncRegistriesTransactionMutex.acquire();
16551668

1656-
const transaction = await sequelize.transaction();
1669+
// Release both mutexes exactly once on every exit path.
1670+
let mutexesReleased = false;
1671+
const releaseMutexes = () => {
1672+
if (mutexesReleased) {
1673+
return;
1674+
}
1675+
mutexesReleased = true;
1676+
releaseAddDeleteMutex();
1677+
releaseAuditTransactionMutex();
1678+
};
1679+
1680+
// Create the transaction inside the try so a failure to open it still
1681+
// routes through releaseMutexes() rather than leaking the locks.
1682+
let transaction;
16571683
try {
1684+
transaction = await sequelize.transaction();
16581685
await Organization.destroy({ where: { orgUid }, transaction });
16591686

16601687
for (const modelKey of Object.keys(ModelKeys)) {
16611688
await ModelKeys[modelKey].destroy({ where: { orgUid }, transaction });
16621689
}
16631690

1664-
await Staging.truncate({ transaction });
1691+
if (!skipStagingTruncate) {
1692+
await Staging.truncate({ transaction });
1693+
}
16651694
await FileStore.destroy({ where: { orgUid }, transaction });
16661695
await Audit.destroy({ where: { orgUid }, transaction });
16671696

16681697
await transaction.commit();
1669-
1670-
await Meta.addUserDeletedOrgUid(orgUid);
16711698
} catch (error) {
16721699
logger.error(
16731700
`failed to delete all db records for organization ${orgUid}, rolling back changes. Error: ${error.message}`,
16741701
);
1675-
await transaction.rollback();
1702+
// Defensive rollback: a throwing rollback must not bypass the mutex
1703+
// release and leak the locks. transaction may be undefined if it failed
1704+
// to open.
1705+
try {
1706+
if (transaction) {
1707+
await transaction.rollback();
1708+
}
1709+
} catch (rollbackError) {
1710+
logger.error(
1711+
`[v1]: rollback failed while deleting organization ${orgUid}: ${rollbackError.message}`,
1712+
);
1713+
}
1714+
releaseMutexes();
16761715
throw new Error(
16771716
`an error occurred while deleting records corresponding to organization ${orgUid}. no changes have been made`,
16781717
);
1718+
}
1719+
1720+
// Record the deletion after a successful commit, while still holding the
1721+
// mutexes so the default-org sync cannot re-import the org between the
1722+
// commit and the meta write. Keep the meta write out of the transaction
1723+
// try so a post-commit failure never triggers a rollback of an already
1724+
// committed transaction.
1725+
try {
1726+
if (recordUserDeleted) {
1727+
await Meta.addUserDeletedOrgUid(orgUid);
1728+
}
16791729
} finally {
1680-
releaseAddDeleteMutex();
1681-
releaseAuditTransactionMutex();
1730+
releaseMutexes();
16821731
}
16831732
}
16841733

‎src/models/v2/organizations-v2.model.js‎

Lines changed: 75 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -2151,11 +2151,23 @@ class OrganizationsV2 extends Model {
21512151
}
21522152

21532153
/**
2154-
* Delete all V2 data for an organization
2154+
* Delete all V2 data for an organization: the org's full registry data (via
2155+
* purgeV2OrganizationData) plus its organization and audit rows.
21552156
* @param {string} orgUid - Organization UID
2156-
* @returns {Promise<void>}
2157+
* @param {object} [options]
2158+
* @param {boolean} [options.recordUserDeleted=true] - When true, the org is
2159+
* recorded in the meta user-deleted list so default-org sync will not
2160+
* re-import it. Background orglist reconcile passes `false`: an org removed
2161+
* because it left the governance orgList is not a user deletion and must not
2162+
* be suppressed if it is later re-added or ONLY_CADT_SUBSCRIPTIONS is
2163+
* disabled.
2164+
* @param {number} [options.retryCount=0] - Internal: SQLITE_BUSY retry depth.
2165+
* @returns {Promise<number>} total number of registry rows purged
21572166
*/
2158-
static async deleteAllOrganizationData(orgUid, retryCount = 0) {
2167+
static async deleteAllOrganizationData(
2168+
orgUid,
2169+
{ recordUserDeleted = true, retryCount = 0 } = {},
2170+
) {
21592171
const maxRetries = 10;
21602172
const baseDelay = 200; // 200ms base delay
21612173
const maxDelay = 5000; // 5 seconds max delay
@@ -2170,9 +2182,34 @@ class OrganizationsV2 extends Model {
21702182
const releaseAuditTransactionMutex =
21712183
await processingSyncRegistriesTransactionMutexV2.acquire();
21722184

2173-
const transaction = await sequelizeV2.transaction();
2185+
// Release both mutexes exactly once on every exit path. Without this, a
2186+
// throw after commit (e.g. in the post-commit meta write) would leak both
2187+
// mutexes and permanently wedge all V2 deletes and sync transactions.
2188+
let mutexesReleased = false;
2189+
const releaseMutexes = () => {
2190+
if (mutexesReleased) {
2191+
return;
2192+
}
2193+
mutexesReleased = true;
2194+
releaseAddDeleteMutex();
2195+
releaseAuditTransactionMutex();
2196+
};
2197+
2198+
let purgedRowCount;
2199+
// Create the transaction inside the try so a failure to open it (e.g.
2200+
// SQLITE_BUSY / pool exhaustion) still routes through releaseMutexes().
2201+
let transaction;
21742202
try {
2203+
transaction = await sequelizeV2.transaction();
21752204
const { AuditV2 } = await import('./index.js');
2205+
const { purgeV2OrganizationData } = await import(
2206+
'../../utils/v2-org-data-purge.js'
2207+
);
2208+
2209+
// Remove every registry record this org created (projects, units and all
2210+
// project-scoped child records). No reference guards are applied: the
2211+
// org's data is purged unconditionally.
2212+
purgedRowCount = await purgeV2OrganizationData(orgUid, { transaction });
21762213

21772214
// Delete from organization table
21782215
await OrganizationsV2.destroy({
@@ -2184,15 +2221,26 @@ class OrganizationsV2 extends Model {
21842221
// Staging is temporary and will be cleared on next commit cycle
21852222
// We skip truncating staging here to avoid database locks
21862223

2187-
// Delete from audit table (only V2 data model with org_uid)
2224+
// Delete from audit table
21882225
await AuditV2.destroy({
21892226
where: { org_uid: orgUid },
21902227
transaction,
21912228
});
21922229

21932230
await transaction.commit();
21942231
} catch (error) {
2195-
await transaction.rollback();
2232+
// Roll back defensively: a throwing rollback (e.g. on an already-closed
2233+
// connection) must not bypass the mutex releases below and leak the locks.
2234+
// transaction may be undefined if it failed to open.
2235+
try {
2236+
if (transaction) {
2237+
await transaction.rollback();
2238+
}
2239+
} catch (rollbackError) {
2240+
loggerV2.error(
2241+
`[v2]: rollback failed while deleting organization ${orgUid}: ${rollbackError.message}`,
2242+
);
2243+
}
21962244

21972245
// Check if it's a database lock error and we haven't exceeded max retries
21982246
// Check both error.message and error.original (Sequelize wraps errors)
@@ -2217,38 +2265,42 @@ class OrganizationsV2 extends Model {
22172265
);
22182266

22192267
// Release mutexes before retry
2220-
releaseAddDeleteMutex();
2221-
releaseAuditTransactionMutex();
2268+
releaseMutexes();
22222269

22232270
// Wait before retry
22242271
await new Promise((resolve) => setTimeout(resolve, delay));
22252272

22262273
// Retry the operation
2227-
return await OrganizationsV2.deleteAllOrganizationData(
2228-
orgUid,
2229-
retryCount + 1,
2230-
);
2274+
return await OrganizationsV2.deleteAllOrganizationData(orgUid, {
2275+
recordUserDeleted,
2276+
retryCount: retryCount + 1,
2277+
});
22312278
}
22322279

22332280
// If not a lock error or max retries exceeded, throw the error
22342281
loggerV2.error(
22352282
`[v2]: failed to delete all db records for organization ${orgUid}, rolling back changes. Error: ${error.message}`,
22362283
);
2237-
releaseAddDeleteMutex();
2238-
releaseAuditTransactionMutex();
2284+
releaseMutexes();
22392285
throw new Error(
22402286
`an error occurred while deleting records corresponding to organization ${orgUid}. no changes have been made`,
22412287
);
22422288
}
2243-
// Record the deletion in the meta table while still holding the mutexes so
2244-
// sync-default-organizations-v2 cannot slip in between the delete and the
2245-
// meta write and re-import the org. addUserDeletedOrgUid does not acquire
2246-
// either mutex, so this cannot deadlock.
2247-
const { MetaV2: MetaV2Post } = await import('./index.js');
2248-
await MetaV2Post.addUserDeletedOrgUid(orgUid);
2249-
2250-
releaseAddDeleteMutex();
2251-
releaseAuditTransactionMutex();
2289+
2290+
try {
2291+
// Record the deletion in the meta table while still holding the mutexes so
2292+
// sync-default-organizations-v2 cannot slip in between the delete and the
2293+
// meta write and re-import the org. addUserDeletedOrgUid does not acquire
2294+
// either mutex, so this cannot deadlock.
2295+
if (recordUserDeleted) {
2296+
const { MetaV2: MetaV2Post } = await import('./index.js');
2297+
await MetaV2Post.addUserDeletedOrgUid(orgUid);
2298+
}
2299+
} finally {
2300+
releaseMutexes();
2301+
}
2302+
2303+
return purgedRowCount;
22522304
}
22532305

22542306
/**

‎src/tasks/sync-default-organizations-v2.js‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ import { loggerV2 } from '../config/logger.js';
99
import { getConfig, getConfigV2 } from '../utils/config-loader.js';
1010
import {
1111
buildOrgListAllowSet,
12-
unsubscribeOrgsNotInOrgList,
12+
removeOrgsNotInOrgList,
1313
} from '../utils/orglist-subscription-reconcile.js';
1414

1515
const CONFIG = getConfig().APP;
@@ -123,7 +123,7 @@ const task = new Task('sync-default-organizations-v2', async () => {
123123
if (onlyCadtSubscriptions) {
124124
const { GOVERNANCE_BODY_ID } = getConfigV2().GOVERNANCE;
125125
const allowSet = buildOrgListAllowSet(defaultOrgList, GOVERNANCE_BODY_ID);
126-
await unsubscribeOrgsNotInOrgList({
126+
await removeOrgsNotInOrgList({
127127
defaultOrgList,
128128
allowSet,
129129
organizationModel: OrganizationsV2,
@@ -134,6 +134,12 @@ const task = new Task('sync-default-organizations-v2', async () => {
134134
},
135135
unsubscribeFromOrganizationStores:
136136
OrganizationsV2.unsubscribeFromOrganizationStores.bind(OrganizationsV2),
137+
// Background removal of an off-orglist org is not a user deletion, so
138+
// it must not be recorded in the user-deleted suppression list.
139+
deleteAllOrganizationData: (orgUid) =>
140+
OrganizationsV2.deleteAllOrganizationData(orgUid, {
141+
recordUserDeleted: false,
142+
}),
137143
logger: loggerV2,
138144
apiVersionLabel: 'v2',
139145
});

‎src/tasks/sync-default-organizations.js‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ import { logger } from '../config/logger.js';
99
import { getConfig } from '../utils/config-loader.js';
1010
import {
1111
buildOrgListAllowSet,
12-
unsubscribeOrgsNotInOrgList,
12+
removeOrgsNotInOrgList,
1313
} from '../utils/orglist-subscription-reconcile.js';
1414

1515
const CONFIG = getConfig();
@@ -112,7 +112,7 @@ const task = new Task('sync-default-organizations', async () => {
112112
if (onlyCadtSubscriptions) {
113113
const { GOVERNANCE_BODY_ID } = CONFIG.GOVERNANCE;
114114
const allowSet = buildOrgListAllowSet(defaultOrgRecords, GOVERNANCE_BODY_ID);
115-
await unsubscribeOrgsNotInOrgList({
115+
await removeOrgsNotInOrgList({
116116
defaultOrgList: defaultOrgRecords,
117117
allowSet,
118118
organizationModel: Organization,
@@ -123,6 +123,14 @@ const task = new Task('sync-default-organizations', async () => {
123123
},
124124
unsubscribeFromOrganizationStores:
125125
Organization.unsubscribeFromOrganizationStores.bind(Organization),
126+
// Background removal of a remote org must not wipe the home org's
127+
// pending staged changes (global, un-scoped staging table), and is
128+
// not a user deletion so it must not populate the suppression list.
129+
deleteAllOrganizationData: (orgUid) =>
130+
Organization.deleteAllOrganizationData(orgUid, {
131+
skipStagingTruncate: true,
132+
recordUserDeleted: false,
133+
}),
126134
logger,
127135
apiVersionLabel: 'v1',
128136
});

‎src/utils/defaultConfig.js‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,8 @@ export const defaultConfig = {
3131
AUTO_MIRROR_EXTERNAL_STORES: true,
3232
/**
3333
* When true, DataLayer subscriptions are kept in sync with the governance
34-
* orgList: orgs removed from the list are unsubscribed (data retained);
34+
* orgList: orgs removed from the list are unsubscribed AND fully deleted
35+
* (the org record plus all of its registry data, with no reference checks);
3536
* orgs on the list with subscribed=false are re-subscribed. Requires a
3637
* non-empty orgList from a successful governance sync before any removal.
3738
*/

0 commit comments

Comments
 (0)