feat: Use S3 node store with seaweedfs by BYK · Pull Request #3498 · getsentry/self-hosted

BYK · 2024-12-31T13:18:43Z

Enables S3 node store using SeaweedFS and sentry-nodestore-s3 by @stayallive

This should alleviate all the issues stemming from (ab)using PostgreSQL as the node store.

codecov · 2024-12-31T13:54:45Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.49%. Comparing base (440b658) to head (95a4de2).
⚠️ Report is 1 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3498   +/-   ##
=======================================
  Coverage   99.49%   99.49%           
=======================================
  Files           3        3           
  Lines         197      197           
=======================================
  Hits          196      196           
  Misses          1        1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

aldy505 · 2024-12-31T14:47:08Z

Any reason why you didn't use SeaweedFS per what you said yesterday?

garage.toml

install/bootstrap-garage.sh

docker-compose.yml

sentry/sentry.conf.example.py

install/bootstrap-garage.sh

Co-authored-by: Reinaldy Rafli <aldy505@proton.me>

docker-compose.yml

BYK · 2024-12-31T21:08:59Z

@aldy505

Any reason why you didn't use SeaweedFS per what you said yesterday?

Well I started with that and realized 3 things:

It really is not geared towards single-node setups and have nodes with different roles. This makes is more challenging to scale up or set up in our setup
It has this paid admin interface. Not a deal breaker but it is clear that it is geared towards more "professional" setups
Its S3 API interface support is not really great

Garage fits the bill much better as it is explicitly created for smaller setups like this, easy to expand without specialized roles, doesn't have any paid thing in it, and has much more decent and familiar S3 interface support.

doc-sheet · 2025-05-25T09:29:51Z

It really is not geared towards single-node setups and have nodes with different roles. This makes is more challenging to scale up or set up in our setup

when I tried seaweedfs last time (and I still use it for sourcemap/profile storage tbh) it had single node ability via weed server command.
Like

weed server -filter=true -s3=true -master=true -volume=true

Some of them enabled by default.

doc-sheet · 2025-05-25T09:33:02Z

I think garage/minio simpler for small setups, seaweedfs looks necessary for mid to high setups because all other services I know keep files as is.

And thousands of thousands small files like profiles not ideal to store on most popular filesystems i guess.

aldy505 · 2025-06-04T01:24:30Z

I think garage/minio simpler for small setups, seaweedfs looks necessary for mid to high setups because all other services I know keep files as is.

@doc-sheet Hey, I'm going to work on this PR. I'd think seaweed is better for self-hosted Sentry. One thing I don't like about Garage is that we need to specify the storage allocation beforehand, if we set it to 100GB, there might be some people that have more data than 100GB, I don't want that to cause any issues.

That said, since you said you've used seaweed before: How was your experience? How does it compare to MinIO or Ceph?

And thousands of thousands small files like profiles not ideal to store on most popular filesystems i guess.

Yeah if we set up an object storage, we might as well move filestore & profiles there too. But let's focus on nodestore first.

doc-sheet · 2025-06-04T07:23:55Z

How was your experience? How does it compare to MinIO or Ceph?

It is a bit strange sometimes. But it is fine.

It has multiple options for filer store.
I didn't try leveldb storage aiming to fault tolerance.

At first I tried redis it worked for several months and then... I just lost all data.
It was there physically but wasn't available from API (s3 or web) - each list call returned different results.

I don't know if issue was in redis or weed itself. i suspect bug with ttl could be the reason too.

But after that incident I wiped cluster and started new one with scylla as a filer backend and it works fine for almost a year already despite that ttl bug.

Seaweedfs have multiple versions like

3.89
3.89_full
3.89_large_disk
3.89_large_disk_full

I suggest to use large_disk always. Documentation is not clear but it is easy to reach that limit
https://github.com/seaweedfs/seaweedfs/wiki/FAQ#how-to-configure-volumes-larger-than-30gb

I don't know difference between full and normal and just use _large_disk_full builds :)

Also I don't use s3 auth - I was too lazy to set it up.

Other than all that I have no problems and barely touched it after initial setup. It just works.
I added some volumes but not removed any yet.

As for minio and ceph.
I never used ceph.

But minio was the reason to look for alternatives.

Tons of profiles from js-sdk stored as different files started to affect my monitoring script and soon it might start to affect minio performance too.

And it is not that easy to scale minio. And probably impossible to optimize for small-files storage. At least in my low-cost setup.

doc-sheet · 2025-06-04T07:31:32Z

let's focus on nodestore first.

If seaweedfs would control ttl then there is another catch.
I'm not sure if it is possible to control ttl with s3-api already.

weed have it's own settings for collections and it creates collection for each s3-bucket.
https://github.com/seaweedfs/seaweedfs/wiki/S3-API-FAQ#setting-ttl

But if sentry itself would cleanup old data I guess there is no difference.

aldy505 · 2025-06-05T05:49:36Z

How was your experience? How does it compare to MinIO or Ceph?

It is a bit strange sometimes. But it is fine.

It has multiple options for filer store. I didn't try leveldb storage aiming to fault tolerance.

At first I tried redis it worked for several months and then... I just lost all data. It was there physically but wasn't available from API (s3 or web) - each list call returned different results.

I don't know if issue was in redis or weed itself. i suspect bug with ttl could be the reason too.

But after that incident I wiped cluster and started new one with scylla as a filer backend and it works fine for almost a year already despite that ttl bug.

Seaweedfs have multiple versions like

3.89

3.89_full

3.89_large_disk

3.89_large_disk_full

I suggest to use large_disk always. Documentation is not clear but it is easy to reach that limit https://github.com/seaweedfs/seaweedfs/wiki/FAQ#how-to-configure-volumes-larger-than-30gb

I don't know difference between full and normal and just use _large_disk_full builds :)

Also I don't use s3 auth - I was too lazy to set it up.

Other than all that I have no problems and barely touched it after initial setup. It just works. I added some volumes but not removed any yet.

Good to know about Seaweed

As for minio and ceph. I never used ceph.

But minio was the reason to look for alternatives.

Tons of profiles from js-sdk stored as different files started to affect my monitoring script and soon it might start to affect minio performance too.

And it is not that easy to scale minio. And probably impossible to optimize for small-files storage. At least in my low-cost setup.

Ah so everyone has the same experience with minio.

let's focus on nodestore first.

If seaweedfs would control ttl then there is another catch. I'm not sure if it is possible to control ttl with s3-api already.

weed have it's own settings for collections and it creates collection for each s3-bucket. https://github.com/seaweedfs/seaweedfs/wiki/S3-API-FAQ#setting-ttl

But if sentry itself would cleanup old data I guess there is no difference.

The sentry cleanup job only cleans up the one on filesystem. If we're using S3, it won't clean up anything. We need to configure S3 data cleanup on our own.

doc-sheet · 2025-06-06T20:24:16Z

Looks like i missed that seaweedfs now have an ability to control ttl with s3 api. And I even linked to correct section of FAQ. :)

I'd like to look into new integraton with seaweedfs.

Nad by the way I like the idea of expanding sentry images.

I am myself install some packages and modules.

Like maybe an extra step in install to build user provided Dockerfiles.

aldy505 · 2025-09-06T06:44:06Z

Config migration is done and is behind a prompt/flag. Next up is to think how to manage the retention. Should we do it with a cron/scheduled job (cons: probably heavy process, pros: can be configured dynamically), or a S3 lifecycle (cons: can't be configured dynamically, pros: won't be a heavy process).

aldy505 · 2025-09-09T14:20:43Z

I'm just gonna go forward with this https://github.com/seaweedfs/seaweedfs/wiki/S3-API-FAQ#setting-ttl

This reverts commit 2f1575d.

aldy505 · 2025-09-09T15:22:04Z

Great, I believe that's all.

aldy505 · 2025-09-09T15:22:47Z

@aminvakil @doc-sheet Hi, would you mind reviewing this PR?

aldy505

Unblocking myself.

BYK

Arguing with myself 😝 (comments for @aldy505 )

BYK · 2025-09-11T11:07:41Z

install/bootstrap-s3-nodestore.sh

@@ -0,0 +1,90 @@
+echo "${_group}Bootstrapping seaweedfs (node store)..."
+
+$dc up --wait seaweedfs postgres


Does the --wait thing work for podman?

Good point. Will check later.

BYK · 2025-09-11T11:09:14Z

install/bootstrap-s3-nodestore.sh

+
+bucket_list=$($s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' ls)
+
+if [[ $($bucket_list | tail -1 | awk '{print $3}') != 's3://nodestore' ]]; then


This if condition really needs some explanation about what it is checking and how -- like what are the assumptions here?

BYK · 2025-09-11T11:10:29Z

sentry/sentry.conf.example.py

+# Other backend implementations for node storage developed by the community
+# are available in public GitHub repositories.
+
+SENTRY_NODESTORE = "sentry_nodestore_s3.S3PassthroughDjangoNodeStorage"


Do we need Passthrough for fresh installs?

This is your original code, so I assume, yes.

I would not assume anything about my original code 😅

BYK · 2025-09-11T11:11:46Z

install/bootstrap-s3-nodestore.sh

+        read -p "y or n? " yn
+        case $yn in
+        y | yes | 1)
+          export APPLY_AUTOMATIC_CONFIG_UPDATES=1


This feels dangerous as once you export it like this wouldn't it also enable auto update for pgbouncer? I think that should be a separate questions and decision?

Ah good point.

BYK · 2025-09-11T11:12:07Z

install/bootstrap-s3-nodestore.sh

+        n | no | 0)
+          export APPLY_AUTOMATIC_CONFIG_UPDATES=0
+          echo
+          echo -n "Alright, you will need to update your sentry.conf.py file manually before running 'docker compose up'."


Again, provide details and link to this PR (or some doc). Also skip setting up seaweed at this point or at least mention that the service will be running for no good reason?

BYK · 2025-09-11T11:13:32Z

install/bootstrap-s3-nodestore.sh

+
+    if [[ "$APPLY_AUTOMATIC_CONFIG_UPDATES" == 1 ]]; then
+      nodestore_config=$(sed -n '/SENTRY_NODESTORE/,/[}]/{p}' sentry/sentry.conf.example.py)
+      if [[ $($dc exec postgres psql -qAt -U postgres -c "select exists (select * from nodestore_node limit 1)") = "f" ]]; then


Up until this point you don't need the postgres service so maybe instead of bringing it up at the beginning, just use $dcr postgres psql ... here?

BYK · 2025-09-11T11:14:49Z

install/bootstrap-s3-nodestore.sh

+  $dc exec seaweedfs mkdir -p /data/idx/
+  $s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' mb s3://nodestore


Why are these commands repeated (from lines 6 and 7 above)?

BYK · 2025-09-11T11:15:39Z

install/bootstrap-s3-nodestore.sh

+  # XXX(aldy505): Should we refactor this?
+  lifecycle_policy=$(
+    cat <<EOF
+<?xml version="1.0" encoding="UTF-8"?>
+<LifecycleConfiguration>
+    <Rule>
+        <ID>Sentry-Nodestore-Rule</ID>
+        <Status>Enabled</Status>
+        <Filter></Filter>
+        <Expiration>
+            <Days>$SENTRY_EVENT_RETENTION_DAYS</Days>
+        </Expiration>
+    </Rule>
+</LifecycleConfiguration>
+EOF
+  )
+  $dc exec seaweedfs sh -c "printf '%s' '$lifecycle_policy' > /tmp/nodestore-lifecycle-policy.xml"


Yes, make it a config file and mount it? Is this because of the $SENTRY_EVENT_RETENTION_DAYS variable? If yes, we may wanna construct a custom entry point script for seaweed so when people change this value and restart, it is updated without running ./install.sh again?

Restarting seaweed wouldn't take anything into effect. You will need to execute s3cmd setlifecycle for that

And I'm saying we should put whatever is necessary into a custom entrypoint script for seaweed so it picks the changes up?

I don't like that approach, it complicates stuff.

Okay, until someone compalins let's go with the simpler approach :)

BYK · 2025-09-11T11:19:06Z

install/bootstrap-s3-nodestore.sh

+  echo "Making sure the bucket lifecycle policy is all set up correctly..."
+  $s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' getlifecycle s3://nodestore


This seems it belongs to tests instead of here? Is this because of the unknown env variable value?

aminvakil

Can we make it opt-in for a release? And get some feedback regarding this change?

aldy505 · 2025-09-12T00:18:34Z

Can we make it opt-in for a release? And get some feedback regarding this change?

@aminvakil You're saying this shouldn't be the default? And probably switch it as a default at a later stage? That works for me.

BYK · 2025-09-12T07:37:18Z

I'd say default for fresh installs. And since we prompt people for existing installs, it should be fine?

aminvakil · 2025-09-12T11:02:49Z

@aldy505 Yes.

@BYK Yes.

I have zero knowledge about seaweedfs and very limited knowledge about S3 storages, therefore I'd suggest merging this after 15th, so people who use main branch can test it in different environments they have.

But again that's because my limited knowledge of S3 storages and I do not know who big this change is.

aminvakil · 2025-09-12T11:15:10Z

install/bootstrap-s3-nodestore.sh

+EOF
+  )
+  $dc exec seaweedfs sh -c "printf '%s' '$lifecycle_policy' > /tmp/nodestore-lifecycle-policy.xml"
+  $s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' setlifecycle /tmp/nodestore-lifecycle-policy.xml s3://nodestore


What does --region=us-east-1 mean here?

It's the default region name for AWS S3. It doesn't mean anything for self-hosted, but we can only put known region values.

aminvakil · 2025-09-12T11:17:19Z

sentry/sentry.conf.example.py

+    "endpoint_url": "http://seaweedfs:8333",
+    "bucket_path": "nodestore",
+    "bucket_name": "nodestore",
+    "region_name": "us-east-1",


Is region_name mandatory?

Yep. Otherwise it'll complain of missing region name.

aldy505 · 2025-09-12T14:05:37Z

@aldy505 Yes.

@BYK Yes.

I have zero knowledge about seaweedfs and very limited knowledge about S3 storages, therefore I'd suggest merging this after 15th, so people who use main branch can test it in different environments they have.

But again that's because my limited knowledge of S3 storages and I do not know who big this change is.

Great! I'll change it later tonight.

aldy505 · 2025-09-12T14:06:10Z

Uh, I mean lets do what @BYK said. I still prefer to merge all this before 15th.

This reverts commit 84f904f.

BYK added 6 commits December 31, 2024 16:18

feat: Use S3 node store with garage

364455b

lol, fix bash

803bef3

moar bash

58b301c

lol

bd414d5

hate bash

a2ac10d

fix moar bash

7b28dfb

aldy505 reviewed Dec 31, 2024

View reviewed changes

garage.toml Outdated Show resolved Hide resolved

garage.toml Outdated Show resolved Hide resolved

install/bootstrap-garage.sh Outdated Show resolved Hide resolved

docker-compose.yml Outdated Show resolved Hide resolved

sentry/sentry.conf.example.py Show resolved Hide resolved

aldy505 reviewed Dec 31, 2024

View reviewed changes

install/bootstrap-garage.sh Outdated Show resolved Hide resolved

BYK and others added 2 commits December 31, 2024 15:23

Add healthcheck to garage service

4a6d337

Co-authored-by: Reinaldy Rafli <aldy505@proton.me>

revert +x

595ed68

hubertdeng123 reviewed Dec 31, 2024

View reviewed changes

docker-compose.yml Outdated Show resolved Hide resolved

docker-compose.yml Outdated Show resolved Hide resolved

BYK added 2 commits December 31, 2024 23:43

fix healthcheck, fix config

99576bf

add env var for garage size

2a55a40

BYK added 4 commits January 1, 2025 00:22

use better compression level

a29cf0b

simpler garage config

a0ae9b1

Merge branch 'master' into byk/feat/s3-nodestore

20e608b

add migration support

8d7c1ff

aldy505 mentioned this pull request Jan 5, 2025

Cleaning nodestore_node table #1808

Closed

This was referenced Mar 18, 2025

Ongoing Work & Feedback #3625

Open

Redundancy for a self-host installation running on a linux VM on Azure #3626

Closed

aldy505 added 2 commits September 6, 2025 13:35

Merge remote-tracking branch 'origin/master' into byk/feat/s3-nodestore

75a9ca5

feat: nodestore config update behind a prompt/flag

81cde28

aldy505 added 7 commits September 9, 2025 21:20

Merge branch 'master' into byk/feat/s3-nodestore

5aa35db

feat: set s3 lifecycle policy

7eb223c

fix: seaweed is a busybox

fa3fc92

fix: try xml policy

ace9191

fix: go back to simplified json

2f1575d

Revert "fix: go back to simplified json"

5753667

This reverts commit 2f1575d.

chore: reword debug lifecycle policy

27dce85

aldy505 approved these changes Sep 11, 2025

View reviewed changes

BYK commented Sep 11, 2025

View reviewed changes

aminvakil reviewed Sep 11, 2025

View reviewed changes

Merge branch 'master' into byk/feat/s3-nodestore

5369a3d

aminvakil reviewed Sep 12, 2025

View reviewed changes

aldy505 added 2 commits September 13, 2025 07:23

fix: don't pollute APPLY_AUTOMATIC_CONFIG_UPDATES variable

639a35d

Merge branch 'master' into byk/feat/s3-nodestore

95a4de2

aldy505 merged commit 84f904f into master Sep 13, 2025
16 checks passed

aldy505 deleted the byk/feat/s3-nodestore branch September 13, 2025 00:48

xprotocol-bot added a commit to xprotocol-org/self-hosted that referenced this pull request Sep 18, 2025

Revert "feat: Use S3 node store with seaweedfs (getsentry#3498)"

b0345fe

This reverts commit 84f904f.

xprotocol-bot added a commit to xprotocol-org/self-hosted that referenced this pull request Sep 24, 2025

Revert "feat: Use S3 node store with seaweedfs (getsentry#3498)"

dfdb492

This reverts commit 84f904f.

github-actions bot locked and limited conversation to collaborators Sep 28, 2025

		@@ -0,0 +1,90 @@
		echo "${_group}Bootstrapping seaweedfs (node store)..."

		$dc up --wait seaweedfs postgres


		bucket_list=$($s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' ls)

		if [[ $($bucket_list \| tail -1 \| awk '{print $3}') != 's3://nodestore' ]]; then

		$dc exec seaweedfs mkdir -p /data/idx/
		$s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' mb s3://nodestore

		echo "Making sure the bucket lifecycle policy is all set up correctly..."
		$s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' getlifecycle s3://nodestore

Uh oh!

Conversation

BYK commented Dec 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

aldy505 commented Dec 31, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BYK commented Dec 31, 2024

Uh oh!

doc-sheet commented May 25, 2025

Uh oh!

doc-sheet commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aldy505 commented Jun 4, 2025

Uh oh!

doc-sheet commented Jun 4, 2025

Uh oh!

doc-sheet commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aldy505 commented Jun 5, 2025

Uh oh!

doc-sheet commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aldy505 commented Sep 6, 2025

Uh oh!

aldy505 commented Sep 9, 2025

Uh oh!

aldy505 commented Sep 9, 2025

Uh oh!

aldy505 commented Sep 9, 2025

Uh oh!

aldy505 left a comment

Choose a reason for hiding this comment

Uh oh!

BYK left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BYK commented Dec 31, 2024 •

edited

Loading

codecov bot commented Dec 31, 2024 •

edited

Loading

doc-sheet commented May 25, 2025 •

edited

Loading

doc-sheet commented Jun 4, 2025 •

edited

Loading

doc-sheet commented Jun 6, 2025 •

edited

Loading

aldy505 commented Sep 12, 2025 •

edited

Loading