API daemon crashing with invalid memory address or nil pointer dereference #582
Description
We previously had the issue #431
Please let me know if I can provide any details.
Detailed Description
Aptly crashed twice last week, without any real indication whats going on.
We are rebuilding a lot of snapshots over the night.
Jun 8 02:05:09 aptly aptly[16230]: Unable to reopen database, sleeping 10s
Jun 8 02:05:19 aptly aptly[16230]: Unable to reopen database, sleeping 10s
Jun 8 02:05:29 aptly aptly[16230]: Unable to reopen database, sleeping 10s
Jun 8 02:05:39 aptly aptly[16230]: Unable to reopen database, sleeping 10s
Jun 8 02:05:49 aptly aptly[16230]: Unable to reopen database, sleeping 10s
Jun 8 02:05:59 aptly aptly[16230]: Unable to reopen database, sleeping 10s
...
Jun 8 02:06:09 aptly aptly[16230]: panic: runtime error: invalid memory address or nil pointer dereference
Jun 8 02:06:09 aptly aptly[16230]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x168 pc=0x776545]
Jun 8 02:06:09 aptly aptly[16230]: goroutine 7 [running]:
Jun 8 02:06:09 aptly aptly[16230]: github.com/smira/aptly/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).isClosed(0x0, 0x7f70119934b0)
Jun 8 02:06:09 aptly aptly[16230]: /Users/smira/Documents/go/src/github.com/smira/aptly/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:230 +0x5
Jun 8 02:06:09 aptly aptly[16230]: github.com/smira/aptly/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).ok(0x0, 0x7f70119934b0, 0x0)
Jun 8 02:06:09 aptly aptly[16230]: /Users/smira/Documents/go/src/github.com/smira/aptly/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:235 +0x2b
Jun 8 02:06:09 aptly aptly[16230]: github.com/smira/aptly/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).NewIterator(0x0, 0x0, 0x0, 0x0, 0x0)
Jun 8 02:06:09 aptly aptly[16230]: /Users/smira/Documents/go/src/github.com/smira/aptly/vendor/github.com/syndtr/goleveldb/leveldb/db.go:876 +0x47
Jun 8 02:06:09 aptly aptly[16230]: github.com/smira/aptly/database.(*levelDB).FetchByPrefix(0xc4200acae0, 0xc420fde088, 0x1, 0x8, 0x0, 0x0, 0x0)
Jun 8 02:06:09 aptly aptly[16230]: /Users/smira/Documents/go/src/github.com/smira/aptly/database/leveldb.go:172 +0xba
Jun 8 02:06:09 aptly aptly[16230]: github.com/smira/aptly/deb.NewRemoteRepoCollection(0xfdd6c0, 0xc4200acae0, 0xc4201595a8)
Jun 8 02:06:09 aptly aptly[16230]: /Users/smira/Documents/go/src/github.com/smira/aptly/deb/remote.go:657 +0xfa
Jun 8 02:06:09 aptly aptly[16230]: github.com/smira/aptly/deb.(*CollectionFactory).RemoteRepoCollection(0xc42000f9a0, 0x0)
Jun 8 02:06:09 aptly aptly[16230]: /Users/smira/Documents/go/src/github.com/smira/aptly/deb/collections.go:51 +0xa1
Jun 8 02:06:09 aptly aptly[16230]: github.com/smira/aptly/api.flushColections()
Jun 8 02:06:09 aptly aptly[16230]: /Users/smira/Documents/go/src/github.com/smira/aptly/api/api.go:34 +0x3f
Jun 8 02:06:09 aptly aptly[16230]: github.com/smira/aptly/api.acquireDatabase(0xc420011260, 0xc4200112c0)
Jun 8 02:06:09 aptly aptly[16230]: /Users/smira/Documents/go/src/github.com/smira/aptly/api/api.go:94 +0x80
Jun 8 02:06:09 aptly aptly[16230]: created by github.com/smira/aptly/api.Router
Jun 8 02:06:09 aptly aptly[16230]: /Users/smira/Documents/go/src/github.com/smira/aptly/api/router.go:26 +0x196
Jun 8 02:06:09 aptly systemd[1]: aptly-api.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 8 02:06:09 aptly systemd[1]: Unit aptly-api.service entered failed state.
The only API accesses at that file were:
10.10.0.178 - apiuser [08/Jun/2017:02:04:29 +0200] "DELETE /api/files/app-openSUSE-42.3-allarch-snapshot HTTP/1.1" 502 5513 "-" "curl/7.37.0"
10.10.0.178 - apiuser [08/Jun/2017:02:06:09 +0200] "POST /api/files/app-openSUSE-42.3-allarch-snapshot HTTP/1.1" 503 5361 "-" "curl/7.37.0"
We are uploading RPM files here, aptly is only used to upload the files obviously.
Looks like the DELETE crashed the daemon. Theoretically the directory mentioned should be there on deletion, but empty.
We use another script to handle the RPM repositories.
Your Environment
- OS: Debian 8.8
- Aptly: 1.0.0+95+g0d04189 (nightly installed yesterday)
Aptly API is running behind an Apache proxy, securing access.