This is more of an opinion than an issue per say. I have found that the implementation of index sets, as it is in current version (2.2 RC1) is limiting the usability of the feature and in some cases (not many I suppose) making it completely unusable altogether.

Expected Behavior and Context

Handling of the data should from my point of perspective be done right after receiving and parsing the message. Not after routing it to streams. Handling the data and organizing it to streams by datasets (and not allowing one stream to look into more data sets at the same time) is creating conflicting problems with different types of logs stored in different index sets.

Scenario to demonstrate:
I have 2 applications. APP1 and APP2. Both have different people in S2/S3 support and hence I need to differentiate user access. At the same time, both applications have two types of logfiles. Application logs and HTTP logs. HTTP logs are only important for the first 3 days. Application logs have to be kept for 3 months. I would expect that I can create a stream that will include all messages from index sets for application AND HTTP logs (and set permissions accordingly for appropriate teams only) to allow them to corelate HTTP logs with Application logs at the same time.

Current Behavior

I can only allow them to search one type of logs at any one time.
Or I can disable user restrictions and let everyone search in "All" stream (that contains trillions of data that is of no use to them and strains both elasticsearch nodes and graylog server nodes performance-wise).

Possible Solution

From my perspective there are 3 solutions:

Adopt ES 5.X to allow manual clean up (delete by query).
Move data handling process further back, closer to inputs and further from streams.
Allow one stream to use more index sets at the same time (this is probably the fastest solution).

Your Environment

• Graylog Version: 2.2 RC1
• Elasticsearch Version: 2.2
• Operating System: RedHat Linux 6/7
• Browser version: Any