docstore/awsdynamodb panics on iter.Next() #3405
Description
Describe the bug
Calling iter.Next on a docstore/awsdynamodb derived Query instance panics with an "index out of range" error along the lines of:
panic: runtime error: index out of range [0] with length 0
goroutine 98 [running]:
gocloud.dev/docstore/awsdynamodb.(*documentIterator).Next(0x14000056050, {0x101ba36b0?, 0x10258bb40?}, {{0x101b386e0, 0x1400019bea0}, 0x0, {0x101b454e0, 0x1400019bea0, 0x199},\
{0x140004fc808, ...}})
/usr/local/sfomuseum/go-activitypub/vendor/gocloud.dev/docstore/awsdynamodb/query.go:492 +0x260
The Query itself has two "Where" statements bracketing a date range:
https://github.com/sfomuseum/go-activitypub/blob/main/accounts_database_docstore.go#L45-L69
Curiously, this problem only seems to affect one table (accounts). The same code (essentially) run against a different table (deliveries) does not panic:
https://github.com/sfomuseum/go-activitypub/blob/main/deliveries_database_docstore.go#L46-L58
To Reproduce
The code in question is here:
https://github.com/sfomuseum/go-activitypub/blob/main/cmd/counts-for-date/main.go
Specifically:
https://github.com/sfomuseum/go-activitypub/blob/main/stats/counts.go#L44
Which is run like this:
go run cmd/counts-for-date/main.go \
-accounts-database-uri 'awsdynamodb://accounts?partition_key=Id&allow_scans=true®ion={REGION}&credentials={CREDENTIALS}' \
-blocks-database-uri 'awsdynamodb://blocks?partition_key=Id&allow_scans=true®ion={REGION}&credentials={CREDENTIALS}' \
-boosts-database-uri 'awsdynamodb://boosts?partition_key=Id&allow_scans=true®ion={REGION}&credentials={CREDENTIALS}' \
-deliveries-database-uri 'awsdynamodb://deliveries?partition_key=Id&allow_scans=true®ion={REGION}&credentials={CREDENTIALS}' \
-followers-database-uri 'awsdynamodb://followers?partition_key=Id&allow_scans=true®ion={REGION}&credentials={CREDENTIALS}' \
-following-database-uri 'awsdynamodb://following?partition_key=Id&allow_scans=true®ion={REGION}&credentials={CREDENTIALS}' \
-likes-database-uri 'awsdynamodb://likes?partition_key=Id&allow_scans=true®ion={REGION}&credentials={CREDENTIALS}' \
-messages-database-uri 'awsdynamodb://messages?partition_key=Id&allow_scans=true®ion={REGION}&credentials={CREDENTIALS}' \
-notes-database-uri 'awsdynamodb://notes?partition_key=Id&allow_scans=true®ion={REGION}&credentials={CREDENTIALS}' \
-posts-database-uri 'awsdynamodb://ap_posts?partition_key=Id&allow_scans=true®ion={REGION}&credentials={CREDENTIALS}' \
-date {YYYY-MM-DD} \
-verbose
If you need/want to set up all those tables, the code to do so is here:
https://github.com/sfomuseum/go-activitypub/blob/main/cmd/create-dynamodb-tables/main.go
And:
https://github.com/sfomuseum/go-activitypub/tree/main/app/dynamodb/tables/create
For example:
go run -mod vendor cmd/create-dynamodb-tables/main.go \
-dynamodb-client-uri 'awsdynamodb://?region={REGION}&credentials={CREDENTIALS}'
Note that all of these URIs are using a custom awsdynamodb:// scheme. Under the hood all of this code is using the aaronland/go-aws-dynamodb and aaronland/go-aws-session packages in to support a wider range of declarative credentialing options than those offered by the default AWS libraries:
- https://github.com/aaronland/go-aws-dynamodb
- https://github.com/aaronland/go-aws-session
- https://github.com/aaronland/go-aws-session?tab=readme-ov-file#credentials
There are also Makefile targets for running everything locally against a containerizer DynamoDB instance:
Expected behavior
An iterator whose Next method triggers an io.EOF error.
Version
0.37.0
Additional context
Documentation (albeit incomplete) for the application's database design and specifics about the use of gocloud.dev/docstore are here:
- https://github.com/sfomuseum/go-activitypub?tab=readme-ov-file#databases
- https://github.com/sfomuseum/go-activitypub?tab=readme-ov-file#goclouddevdocstore
The really long think-y think-y version is here: