Motivation

Finding rooms based on message content is a requirement which has been around for quite a long time (see e. g. RocketChat/Rocket.Chat#1615). There is even an open PR RocketChat/Rocket.Chat#7628 . However, there are (and will always be) concerns with respect to performance and DB load.
When it comes down to searching huge installations, I believe there's no other way than using an external search engine. Replicating data to them is no issue as existing hooks already exist, but getting it integrated into the results is currently not possible.
In order to

• integrate those results, a hook shall be provided in the spotlight-method which retrieves results asynchronously.
• ensure proper access permissions, all potentially accessible rooms for the current user shall be determined (which should be a fast, index-based query) and passed to the hook
• allow understanding of search results, the spotlight result should be enhanced by additional information why a particular result has been found (e. g. [subscription, public channel, message content])

Proposal for implementation

For flexibility reasons, a modular integration of configurable search providers shall be realized.
Since a database-connection cannot be assumed to be available, all replication shall be performed on the application layer (e. g. using http-requests).

• Provide a package in Rocket.Chat core which exposes an API for registering external search providers and an interface definition to be implemented by them. Follow model of RoomTypes (see registry and the interface definition. The active search provider's interface methods shall be triggered explicitly from the rocket.chat runtime as alternative to (untyped) webhooks. Relevant events: Room created, message sent, user added/removed, permission changed
• Provide a settings group which generates sections with the settings provided by the search providers
• Provide a generic UI template which is the default visualization for the search provider implementation
• Provide a second package with a reference implementation

To be clarified: How to perform an initial load? Most probably, a dedicated Method triggering the proper interface methods for all existing objects is appropriate.

• How should this potentially long-running background-task be executed?
• Does this need a particular authorization?