@elasticsearchmachine run elasticsearch-ci/part-2

Wow, you are so efficient!

Great stuff!

I have 2 questions:

• What's the reason - if I read correctly - not to sort the index on @timestamp 1st and 2nd on _tsid?
• I wonder if it would be beneficial to make dimension hierarchic, meaning instead of true/false you configure an order class and sort is applied in the given order. Data usually has such hierarchies like that, with prefix search this could be used in optimizations.

* What's the reason - if I read correctly - _not_ to sort the index on `@timestamp` 1st and 2nd on `_tsid`?

That's a great question! One who's answer I'll want to stick in the javadocs somewhere. But I'll reply here too. We sort on tsid first because all the things we want to do work better that way. Well, two things I can think of:

1. It makes it very easy to, at search time, iterate on a single time series in timestamp order. You do the traditional heap of iterators - this time the iterator is the reader, the timeseries id, and the timestamp. This makes it much easier to implement certain features like getting the rate of change of a counter.
2. _source compression works better when you group a single time series together tightly. A lot better. This has a fair bit to do with the "shape" of the data that folks put in ES for time series data right now. It contains a lot of "tags" that are mostly constant. I don't have a real example handy, but its stuff like, in data about a volume mounted on a k8s pod we log the pod's IP and creation date and which physical node is hosting it. Stuff that doesn't uniquely identify the data but it is important to be able to see. When you sort by timeseries id first lz4 can do a great job compressing these duplicates.

@imotov I sorted by _tsid, @timestamp, both in ascending order. Is that right for you? I believe for the compression the choice of order doesn't matter. But you've thought a lot more about the features we can build on this. Do they need @timestamp in ascending order? Or is descending order OK? I get the sense that doing @timestamp in descending order is going to help a lot of the traditional ES operations like fetching the last 15 minutes of data. Or last hour.

I wonder if it would be beneficial to make dimension hierarchic, meaning instead of true/false you configure an order class and sort is applied in the given order. Data usually has such hierarchies like that, with prefix search this could be used in optimizations.

I like the simplicity of the binary choice for dimension fields, though I had wondered about this too for the case of one high-cardinality dimension and another low-cardinality dimension since I think we'd generally prefer sorting by the low-cardinality dimension first and then by the high-cardinality dimension. Maybe one way to make this work (in a follow-up PR, this one is complex already) would be to allow configuring an index sort on timeseries indices, but only using dimension fields. And then the timeseries mode would add _tsid and @timestamp as tiebreakers automatically.

I like the simplicity of the binary choice for dimension fields, though I had wondered about this too for the case of one high-cardinality dimension and another low-cardinality dimension since I think we'd generally prefer sorting by the low-cardinality dimension first and then by the high-cardinality dimension. Maybe one way to make this work (in a follow-up PR, this one is complex already) would be to allow configuring an index sort on timeseries indices, but only using dimension fields. And then the timeseries mode would add _tsid and @timestamp as tiebreakers automatically.

I like the simplicity too! Being able to configure index sorting with special rules could do the trick. I think its safe to keep this in mind but experiment more with it later.

@weizijun had an similar feeling proposal: allow the users to specify which dimensions are used in routing. If you don't use a dimension in routing then it is just used for index sorting. It'd be useful to force similar documents even closer together. Though I don't know how space savings it'd buy. @weizijun, do you have some experimental results?

I'd like to throw this out there: Traditionally Elasticsearch routes documents to a particular index based on the _id. You can override that by setting the routing explicitly. We'll hash that instead of the id based routing. Whenever folks do that we warn them that they could unbalance the distribution of the documents. They could end up with shard 1 much bigger than shard 0. This makes Elasticsearch work a lot less well. We assume that shards are fairly similarly sized and have fairly similar cost to search.

This PR will cause more unbalanced shards. For sure. But not as many as just going crazy with routing would. Because it scatters different time series to different nodes randomly. We do expect for some time series to get more documents than others. But we expect there to be many many time series so we expect that in most cases the more chatty time series will get scattered to the different shards evenly.

Now, looping around to the routing proposal: it feels like exactly the kind of thing that'd make it easy to accidentally route the high traffic time series indices to the same shard. That doesn't mean its not a good idea too. But, like giving folks more control over the sort order, its something I want to do later. And carefully. Or not at all.

@imotov I sorted by _tsid, @timestamp, both in ascending order.

That's a good question. Since we will be returning data in the ascending order, it might be easier for us to iterate over the data in the ascending order as well, in this case we don't have to flip it later on.

I don't think ordering of _tsid would make any difference as long as all records with the same _tsid are grouped together somehow, so I don't see any reason for not making it ascending to keep things simple.

I wonder if it would be beneficial to make dimension hierarchic

I agree with @jpountz and @nik9000, things are already very complicated as they are, but I also agree with @hendrikmuhs that indeed data have hierarchical structures. I just want to add a caveat that naturally occurring hierarchies do not always match optimizations.

Let's take for example node_cpu_seconds_total from prometheus node exporter. It has 4 dimensions: cpu, instance, job and mode:

node_cpu_seconds_total{cpu="0", instance="node_exporter:9100", job="nodeexporter", mode="idle"}
node_cpu_seconds_total{cpu="0", instance="node_exporter:9100", job="nodeexporter", mode="iowait"}
node_cpu_seconds_total{cpu="0", instance="node_exporter:9100", job="nodeexporter", mode="irq"}
node_cpu_seconds_total{cpu="0", instance="node_exporter:9100", job="nodeexporter", mode="nice"}
node_cpu_seconds_total{cpu="0", instance="node_exporter:9100", job="nodeexporter", mode="softirq"}
node_cpu_seconds_total{cpu="0", instance="node_exporter:9100", job="nodeexporter", mode="steal"}
node_cpu_seconds_total{cpu="0", instance="node_exporter:9100", job="nodeexporter", mode="system"}
node_cpu_seconds_total{cpu="0", instance="node_exporter:9100", job="nodeexporter", mode="user"}

So a natural hierarchy would be job, instance, cpu and then mode. However, a typical query would be most likely interested in average non idle cpu rate sum(1-rate(node_cpu_seconds_total{mode="idle"}[1m])) so, the most likely optimization would be to filter by mode and then process all data inside, which will be adversary scenario for the most logical hierarchy. Or perhaps if we are interested in only one machine the natural hierarchy would indeed help us.

I think this idea is definitely worth exploring, but I would like to spend more time to work with this type of data before we start working on this sort of optimization. I just don't think we will get it right at the moment.

I don't see any reason for not making it ascending to keep things simple.

I'll drop that into a comment. I think its a thing we could change our minds on in the future if we need to.

I've updated the timings in the PR. I don't have a rally track for this (yet) so I'm taking some fairly rough numbers. Indexing my data takes about 2.6 hours with or without timeseries mode. It takes about 10% longer if you sort of all timensions (2.9 hours) instead of generating the time series id and sorting by that. We like generating the timeseries and anyway because we want to aggregate and route on out - but its nice to know we get some time savings by sorting on it.

Similarly, we get time savings when you force merge down to a single segment. The force merge with just best_compression takes .8 hours. In time series mode it takes 110% longer (1.7 hours). This seems to be dominated by merging the sorted segments. If you just use best_compression the segments aren't sorted at all so this isn't surprising. The force_merge time when you sort on every segment is 2.2 hours. That's almost as long as it takes to index the data with best_compression! Still, these show areas for improvement - merging sorted segments is slower than we'd like.

Finally, I got some new size estimates. Remember, this data uncompressed is 150GB. I keep it compressed with zstd and it clocks in at 18gb. gzipped it is 21gb. When you index it with best_compression a single shard is, like we said above, 39.3gb. That's with the search index, doc values, and the _source compressed with zlib. Doing it in time series mode it is 21.8gb, pretty close to gzip even though we still have the search index and doc values! That's lovely. But we can do better. We will do better. In a follow up.

Pinging @elastic/es-analytics-geo (Team:Analytics)

I've converted this back to a draft because I've discovered some "fun" stuff around recoveries. The trick I do with putting the tsid in to the routing doesn't seem to survive recoveries and the translog. I'm investigating.

I'm not sure what's up with it yet.

I am now sure what's up. Its..... icky. The dynamic mapping runs on the data node and doesn't tell the coordinating node what to do. I think the plan for this PR is accept that its broken and then try and fix it in a follow up. Alternatively, we could disallow dynamic mapping adding dimensions. That might be ok in the short term, but I think folks need some way of dynamically adding dimensions and dynamic mappings are the most sensible way to do it in the framework we have now. So we'd have to get it working sooner or later.

I am now sure what's up. Its..... icky. The dynamic mapping runs on the data node and doesn't tell the coordinating node what to do. I think the plan for this PR is accept that its broken and then try and fix it in a follow up. Alternatively, we could disallow dynamic mapping adding dimensions. That might be ok in the short term, but I think folks need some way of dynamically adding dimensions and dynamic mappings are the most sensible way to do it in the framework we have now. So we'd have to get it working sooner or later.

I've pushed a change to detect when we modify the list of dimensions with a dynamic mapping update and reject the document. That'll have to do in the short term. In the long term we'll have to make it work but that's a bigger thing. Big enough that we talked about taking another approach to this change entirely. But, we believe the the majority of the change required to get dynamic mappings working lines up with the change required to get hard time bounds on indices working - so we're going to do it anyway. Probably.

My plan is to scrounge up more reviews for this ASAP and then get this in and start on the dynamic mapping updates.

For anyone following along from home - it took me a while to get good reviews on this, but we found enough "fun" corner cases involving races with mapping that we're rethinking this. We're unlikely to merge this code as it stands, but fairly likely to use it as an inspiration that will be pretty similar in practice. Just without the races.