alerts: Do not alert on long transactions in codeintel-db#36619
Conversation
|
Codenotify: Notifying subscribers in CODENOTIFY files for diff 36c320a...02f0d68.
|
|
@chrismwendt Is it possible to do batches of smaller transactions in the future? Note that long transactions will affect things like the ability to run schema migrations in a reasonable time (if txns are open for 10s of minutes, it can block lock access if we need to create an index, for example). |
|
Unfortunately, holding long transactions is a fairly deeply-rooted requirement in the current implementation of Rockskip. Specifically, there's a single query which regularly updates ~50% of the symbols in the repo. After thinking about this for ~10 minutes it's still not obvious to me how best to break up the transactions, and actually making the change would probably take days. That's not to say it's impossible, but I'm wondering if we can get away with keeping the long transactions. In which cases does blocking index creation cause problems? The first thing that comes to mind is the 1 hour timeout for DB liveness. AFAICT, a long transaction only blocks index creation on tables that the transaction has written to: thread 1: BEGIN;
thread 1: INSERT INTO table1 (column1) VALUES ('value1');
thread 2: CREATE INDEX CONCURRENTLY index1 ON table1 (column1); <-- 🚫 Blocked by thread 1
thread 1: COMMIT;thread 1: BEGIN;
thread 1: INSERT INTO table1 (column1) VALUES ('value1');
thread 2: CREATE INDEX CONCURRENTLY index2 ON table2 (column1); <-- ✅ Not blocked by thread 1
^^^^^^ different table: "table2"
thread 1: COMMIT; |
efritz
left a comment
There was a problem hiding this comment.
@chrismwendt Could you add a comment pre-merge about this being disabled because of rockskip processing? Then if we ever are able to reduce txn time we can revisit this alert.
During normal Rockskip indexing, some transactions that update big hops can take tens of minutes to finish. It would be nice if we could selectively ignore Rockskip transactions while still alerting on other
codeintel-dbtransactions. Any idea how to do that easily? This PR omitscodeintel-dbfrom the check to at least stop the spurious alerts.From https://github.com/sourcegraph/customer/issues/976#issuecomment-1145369932
Test plan
Ran in Grafana