The SPRB usage report does no currently scan for package downloads, this causes the report to miss reporting packages usage and therefore causes false positives in the unreferenced packages list.

A good example to validate is the usage of the microsoft.private.intellisense text-only package.