Skip to content

Extreme slow updates #560

@volker-fr

Description

@volker-fr

Describe the bug
slackdump resume in version 3.1.7 is extremely slow and behaves differently as updates in version 3.0.5

Authentication type (please read) N/A

To Reproduce
Steps to reproduce the behavior:

slackdump 3.0.5

# note that this is 1month back
time ./slackdump \
        export \
        -o existing_folder  \
        -type standard \
        -workspace my_slack \
        -time-from $(date -d "1 month ago" +%Y-%m-%d)T00:00:00"

real    3m55.150s
user    0m6.363s
sys     0m2.038s

slackdump 3.1.7

# note that this is 3 days back only
./slackdump \
        resume \
        -threads \
        -refresh \
        -workspace my_slack \
        -time-from $(date -d "3 days ago" +%Y-%m-%d)T00:00:00 \
        existing_folder

...
... 2025-10-04 03:26:31  ...
...
... 2025-10-04 08:31:51  ...
...
2147.33user 750.46system 5:05:23elapsed 15%CPU (0avgtext+0avgdata 50720maxresident)k
416816inputs+133602024outputs (1major+4276258minor)pagefaults 0swaps

Old slackdump 3.0.5 that doesn't use sqlite catches up 1month in under 4 min. While the newer version 3.1.7 takes over 5 hours for the last three days.

This is a Slack that is limited to 90 days. In a Slack with history far back, the update still runs after over one day.

Expected behavior
Update/refresh of latest Slack messages -- including threads -- has a comparable time between v.3.0.5 and v.3.1.7.

Desktop (please complete the following information):

  • OS: Linux

  • OS Version: Linux playground 6.8.0-85-generic #85-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep 18 15:26:59 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

  • VM but utilizing nvme storage, no issue with memory or CPU, general load looks also fine

  • Slackdump Version: 3.1.7

Additional context
It is unclear what causes this issue.

One surprising part is the delay between updates per log like the following from v3.1.7 slackdump resume --threads ...:

2025-10-05 15:40:06 INFO  stream result: <XXXX>
2025-10-05 15:40:19 INFO  stream result: <Thread[AAA]>
2025-10-05 15:40:24 INFO  stream result: <Thread[BBB]>
2025-10-05 15:40:36 INFO  stream result: <Thread[CCC]>
2025-10-05 15:40:41 INFO  stream result: <Thread[DDD]>

This reminds me about a slowness when importing Slackdump to sqlite from v3.0.5 via slackdump convert -f database ./myworkspace/.

This migration took a really long time. More surprising, everything was on disk and convert doesn't need to reach out to the Slack API itself. Even putting everything into a memory disk for testing didn't speed it up. The slowness was also between messages imported.

If I remember correctly, this showed also up on initial import of a new import from scratch. At that time I enabled debug mode and there was no throttling, just multiple seconds between messages. I checked then also the newly sqlite DB and looked fine with WAL enabled etc.

The surprising part is that none of that was observed in v3.0.5

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions