Skip to content

Feature: Fast member-only channel listing using Edge API #598

@ChrisEdwards

Description

@ChrisEdwards

Problem

For organizations with thousands of Slack channels, the -member-only flag is extremely slow because it still fetches ALL workspace channels before filtering.

Real-world test evidence

Ran a test with -member-only flag on an enterprise workspace:

time slackdump archive -member-only -time-from "2025-01-20" -time-to "2025-01-21" -o test_member_archive

Result: After 6+ minutes, the command was still iterating through ALL workspace channels (not member channels). The output showed sequential channel IDs being processed one by one:

  • C02FSNK0807
  • C02FSRGPJH5
  • C02GAC06DTR
  • C02GJAMV6LE
  • ... hundreds more non-member channels

The command was cancelled after 6 minutes - it was nowhere near complete.

Comparison:

Metric Value
User's member channels ~350
Channels processed in 6 min 200+ (still going)
Expected with fast mode Complete in seconds

Root Cause

The Edge client's GetConversationsContext() in internal/edge/slacker.go runs three parallel operations:

var pipeline = []func(){
    func() {
        // getting client.userBoot information (FAST - YOUR channels only)
        ub, err := cl.ClientUserBoot(ctx)
        // ...
    },
    func() {
        // collecting the IMs (FAST - YOUR DMs only)
        ims, err := cl.IMList(ctx)
        // ...
    },
    func() {
        // collecting the channels (SLOW - ALL workspace channels!)
        ch, err := cl.SearchChannels(ctx, "")  // <-- THE BOTTLENECK
        // ...
    },
}

The SearchChannels(ctx, "") call in internal/edge/search.go:85 has SearchOnlyMyChannels: false, causing it to paginate through EVERY channel in the workspace.

The -member-only flag only filters results AFTER all channels are fetched (internal/chunk/control/processors.go:158):

if c.memberOnly && (ch.ID[0] == 'C' && !ch.IsMember) {
    // skip public non-member channels
    continue
}

Evidence: Fast Endpoints Already Exist

The Edge API already provides fast endpoints that return only the user's channels:

client.userBoot endpoint

  • Returns all channels the user is a member of with full metadata
  • Single API call, returns instantly
  • Tested: Returns 418 channels for a user in a workspace with thousands of channels

client.counts endpoint

  • Returns channels with activity metadata: Latest timestamp, HasUnreads, MentionCount
  • Single API call
  • Tested: Returns 198 channels with activity data

Test script results (same workspace)

ClientCounts returned 85 channels + 68 DMs + 45 MPIMs = 198 total (FAST)
ClientUserBoot returned 418 channels (with full metadata)

Both calls complete in under 1 second. The current -member-only implementation takes 6+ minutes and still doesn't finish.

Proposed Solution

When -member-only is specified, skip the slow SearchChannels("") call entirely. The data from ClientUserBoot() already contains all member channels with full metadata including:

  • Channel ID, name, archived status
  • IsMember flag (always true since it only returns member channels)
  • Topic, purpose, creator
  • Members list

Implementation approach

In internal/edge/slacker.go, modify GetConversationsContext() to accept a "member-only" parameter. When true:

  1. Skip the SearchChannels(ctx, "") goroutine entirely
  2. Rely only on ClientUserBoot() and IMList() which already return member channels

Alternatively, the SearchChannels function already has a SearchOnlyMyChannels field (line 53 in search.go) that could be set to true for member-only mode, but skipping the call entirely would be even faster.

Additional Enhancement: Activity-based filtering

The ClientCounts() endpoint returns activity metadata that could enable powerful new filtering options:

type ChannelSnapshot struct {
    ID           string
    LastRead     fasttime.Time  // When user last read the channel
    Latest       fasttime.Time  // Timestamp of latest message  
    MentionCount int            // Unread @mentions
    HasUnreads   bool           // Has unread messages
}

This could enable flags like:

  • --with-activity - Only channels with any activity
  • --activity-since DATE - Only channels with messages since a date
  • --with-mentions - Only channels with unread @mentions

Use Case

Daily export scripts that archive Slack conversations. Currently users must either:

  1. Hardcode channel IDs (tedious, misses new channels)
  2. Use -member-only which is too slow for large workspaces (6+ minutes and counting)
  3. Wait for the full channel list to complete

With fast member-only mode, users could efficiently archive all their channels without specifying each one manually.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions