Ability to persist CT indexes and resume from the saved index file#49

Merged

d-Rickyy-b merged 17 commits intod-Rickyy-b:masterfrom

messede-degod:feat/ct-index-save-restore

Aug 1, 2025

Contributor

messede-degod commented Jan 14, 2025

Allows CT index to persisted.
CT indexes are dumped to disk every 30 seconds
Future sessions resume from last saved CT index.

d-Rickyy-b reviewed

View reviewed changes

internal/certificatetransparency/ct-watcher.go Outdated

d-Rickyy-b reviewed

View reviewed changes

internal/certificatetransparency/ct-watcher.go

d-Rickyy-b reviewed

View reviewed changes

internal/certificatetransparency/ct-watcher.go Outdated

d-Rickyy-b reviewed

View reviewed changes

internal/certificatetransparency/logmetrics.go Outdated

d-Rickyy-b reviewed

View reviewed changes

internal/certificatetransparency/logmetrics.go Outdated

d-Rickyy-b reviewed

View reviewed changes

internal/certificatetransparency/logmetrics.go Outdated

d-Rickyy-b reviewed

View reviewed changes

internal/config/config.go Outdated

Owner

d-Rickyy-b commented Jan 15, 2025

Thank you for your contribution :) I appreciate it a lot. I left some comments on the code. Feel free to reply and/or fix.

Contributor Author

messede-degod commented Jan 16, 2025

I've addressed all your comments, please do take a look and let me know what you think.

messede-degod changed the title ~~Ability to persist CT indexes and resume from the save index file~~ Ability to persist CT indexes and resume from the saved index file

Owner

d-Rickyy-b commented Feb 11, 2025

Quick update: I did not forget about this PR, I just had a lot of life going on. Tonight I thought about this PR a bit, and some things need a few more thoughts.

start_at_latest_sth and resume_from_ct_index_file should be mutually exclusive, shouldn't they? So why have two variables?
There should be a final save on exit. For that we need some signal handling (Gracefully shutdown certstream-server-go #50). I'll add that in a separate commit on master. For now let's just assume that the Stop() function of the ct-watcher gets called on a graceful shutdown
The index file should be auto-created if it doesn't exist yet
If no setting is defined, the default should be to use the latest STH

I'll add these missing features asap

messede-degod and others added 14 commits

May 4, 2025 17:38


          add: ability to persist and resume from CT indexes

9c0398d


          update: disable CT index persistenc by default

fd8b87c


          update: refactor code, add start_at_latest_sth

cb210ad


          fix: race in GetAllCTIndexes()

bb52131


          update: remove unnecessary index log

eb969db


          chore: add temp files and ct_index.json to gitignore

6887cdb


          refactor: use better names for recovery behavior

ae57b02


          refactor: add proper error handling

9386c60


          refactor: rename fileName to filePath


          refactor: pass ctIndexFilePath to LoadCTIndex function

ac74794


          refactor: create ctIndexFile if not exists

37269b7


          refactor: use new config variables

ed649ea


          refactor: extract saving file into own method

44bd15c


          feat: add feature to create initial index file

4bfea45

This is useful in order to not start pulling each log after the first run at index 0 but still have newly discovered logs download from 0.

d-Rickyy-b force-pushed the feat/ct-index-save-restore branch from 9d6d11f to 4bfea45 Compare

May 5, 2025 00:38

d-Rickyy-b added 3 commits

May 5, 2025 02:40


          chore: add missing trailing newline to gitignore

ef38487


          docs: describe recovery feature in changelog

ab49770


          refactor: remove nested loop and initialize metrics with worker

d646312

Owner

d-Rickyy-b commented May 5, 2025

Sorry for the force push. Was the easiest way to get the rebase done. This PR should now be ready to be merged. Mind taking a look at the new changes and stating your opinion? Thanks a lot!

Contributor Author

messede-degod commented May 14, 2025

looks great, but i have to admit that it looks nothing like what it started off as 😄 .

d-Rickyy-b merged commit bf24abb into d-Rickyy-b:master

3 checks passed

messede-degod mentioned this pull request

Save the last processed index of the certstream log when certstream-server-go exits #46

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet