Proposal: query path tenancy by douglascamata · Pull Request #6320 · thanos-io/thanos

douglascamata · 2023-04-26T16:50:00Z

Yet another proposal for including multi-tenancy support in the query path. In the same base as outlined previously by #4055 (with implementation available at #4141).

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

saswatamcode

Thanks for great work on this proposal! This is definitely something we want to enable, and would be a cool feature to have! 🙂

Some questions/suggestions!

docs/proposals-accepted/202304-query-path-tenancy.md

docs/img/query-path-tenancy-proposal-diagram.svg

Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

docs/proposals-accepted/202304-query-path-tenancy.md

fpetkovski · 2023-04-27T12:18:59Z

docs/proposals-accepted/202304-query-path-tenancy.md

+* Allow the query path components to be configurable to identify tenants, opening the way to the implementation per-tenant features on the query path. These features include, but aren't limited to, the following:
+  * Per-tenant observability.
+  * Per-tenant settings. For example, having different limits per tenant, which is a common request.
+  * Enforce presence of one or more tenant labels in queries.


We can achieve these goals by having separate query instances for individual tenants. A valid goal would be simplifying management for multi-tenant setups and pooling resources to reduce waste.

Don't see any mention of reusing some cortex components that already implement multi-tenancy using an HTTP header. 😦

I wouldn't recommend putting cortex as a dependency. But why not just copy the bits you like here?

we would be happy be deleting code from cortex if it makes it in Thanos 😄

Hey @friedrichg, can you give us some good pointers for this implementation on the Cortex side? I'm not familiar with Cortex and its features, so I'm very unaware of how it implements things.

If it aligns well with how we want to move forward with this, it'll be great to reuse some code (be it importing or copying) so that both projects benefit from it. I know there are some design differences between the two projects, so let's see what can be done and reused.

Thanos and Cortex are very similar, both CNCF projects, sort of rivals in the old days. But nowadays Cortex just reuses a bunch of components from thanos. The compactor, the store-gateway, etc. There is too many things in common.

The thing cortex added from the beginning was the multi-tenancy, so we it adds header to all requests, so all components know what tenant is it about.

And like I said importing cortex into thanos is not a good idea, because it was a thing that happened in the past that created circular dependencies. I think is better you just copy what you need and we reuse it in Cortex (As we already do 😃 )

https://cortexmetrics.io/docs/architecture/

Find me in slack cncf #cortex if you want to chat more about it.

Awesome. I'll get back to this next week and reach out if I have questions or ideas. Thanks. 🙇

Yea, I think it's good point to mention Cortex way of dealing with this as inspiration, but there is nothing wrong in revisiting this feature and checking if there is anything we could improve before implementing.

Ideally, we could have the same solution, so the tools that works on top of Cortex will work on top of Thanos. That's fair advantage to adopting similar patterns. Another advantage is easier migrations between projects, helpful if we will try to unify projects at some point.

Can we mention this somewhere in the doc (advantage and Cortex pattern)? Thanks!

docs/proposals-accepted/202304-query-path-tenancy.md

Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

…nancy-proposal

fpetkovski · 2023-05-02T12:22:45Z

docs/proposals-accepted/202304-query-path-tenancy.md

+## Action Plan
+
+1. Query Frontend
+   1. Implement the `--query-frontend.tenant-header` (default is `""`") and `--query-frontend.default-tenant` (default is `"default-tenant"`) flags. Forward the tenant header to all the configured downstream query endpoints, if it is present.


Configuring the tenant in multiple places can be a big source of confusion for users. Can query frontend simply propagate all headers? I would prefer it we enabled tenancy on the query path using one flag in one component.

We need it to be configurable in multiple places because this header has to go all the way down to the Thanos Store, not only from Query Frontend to Query. It also has to be transmitted from Querier to Store Gateway in setups with, but also without, the Query Frontend and from Querier to Querier in setups with multiple Querier trees.

We need every component in the query path to be able to identify the header containing the tenant name, so there's no escape configuring this everywhere. If we configure it only in the Query Frontend, how does the Querier know which header includes the tenant's name?

I'm against doing a blanket forwarding of all headers. It seems like a potential source of security issues or unknown unintended consequences. WDYT?

Basically, every component has to agree on which header is being used to identify the tenant and we have nothing to facilitate centralized configuration at the moment.

Our only option is to hardcode the tenant header name to be always the same and impose it on all the users of Thanos that want to use this feature.

There would still be the need to enable a flag in the Query Frontend and Queriers.

If we hardcode the header name, we could even be compatible with Cortex and Mimir by only using X-Scope-OrgID.

The way I see it, the querier is the only component that needs to know which header is used for tenancy. It can then take the value of the header and add it to a store api request.

Actually now that I think about this, label injection would happen in the Querier, so only the Querier needs a flag to indicate the tenant label. If we go with @bwplotka's proposal to use pre-defined header names, Query Frontend just needs to forward the header to the Querier, so we won't need additional configuration options.

@fpetkovski as we spoke yesterday, I think we need customizable header name because Receive's already offering that (through --receive.tenant-header="THANOS-TENANT") and using two different header names depending on whether it's a query or a remote write creates confusion through inconsistency.

I think it's valuable to keep the consistency here so that people that already know how Receive works will easily understand how the tenancy enforcement on the query path works.

The default header name will also match Receive's. So we will end up with:

--querier.tenant-header="THANOS-TENANT".

--querier.default-tenant="default-tenant".

--querier.tenancy="false".

When using the default header name, you will have to only add --querier.tenancy="true" to toggle everything on. The simplest case is gracefully simple to handle.

And if you are using a custom header name in Receive, you may (or may not) make the query path consistent with it by customizing --querier.tenant-header. The same applies to the default tenant setting.

cc @bwplotka

As also mentioned by @bwplotka, the possibility of customizing the header name allows us to be more compatible with other tools, like Cortex, which facilitates interoperation and migration between tools.

That makes sense to me.

@fpetkovski actually, we need that header name flag in the Query Frontend too. It cannot simply forward all headers. It needs to know what is the tenant header to include the information in its own metrics and for the per-tenant settings, as previously mentioned in our conversations.

fpetkovski · 2023-05-02T12:27:41Z

docs/proposals-accepted/202304-query-path-tenancy.md

+   2. Implement the `--querier.tenant-label-name` flag. If it is not empty, verify and enforce the tenant label in the query before it is handed to the query engine. As previously mentioned by the proposal, use prom-label-proxy's [Enforce.EnforceMatchers](https://github.com/prometheus-community/prom-label-proxy/blob/main/injectproxy/enforce.go#L141).
+   3. Update the metrics exported by the Querier to include a tenant label with the tenant indicated by the header.
+3. Store Gateway
+   1. Implement the `--store.tenant-header` (default is `""`") and `--query-frontend.default-tenant` (default is `"default-tenant"`) flags. Use the tenant header to identify the tenant in the metrics exported by the Store Gateway.


Can't we use the tenant from request headers?

How do we know the header in the request that includes the tenant's name if the header used to transmit this information can have its name changed?

i.e. Query Frontend configured to use X-User as tenant header. How will Store Gateway know that X-User is the header being used for tenant identification and not anything else?

The querier knows which header is used for tenancy, and it can propagate this header in either the request or in a special gRPC/http header.

This overlaps with our conversation in the other comment, let's solve that one first and then we come back to this one if there will be anything left to discuss. 👍

Also be careful as gRPC does not expose HTTP2 headers - metadata and trailers can be used instead. So flag name for gRPC would need to be different.

@bwplotka aahhh, that's good to know. I'll learn more about it and update the proposal accordingly. Thanks for the tip.

bwplotka

Amazing! Good direction, and some good discussion. I would be supportive of consistent header/metadata, but I would start with hardcoded, opinionated header name, without flags.

I feel for this discussion it would be healthy to explain tenancy model we expect. For example I think read and write tenancy have a little bit different semantics here which are worth to mention.

docs/proposals-accepted/202304-query-path-tenancy.md

bwplotka · 2023-05-03T16:44:35Z

docs/proposals-accepted/202304-query-path-tenancy.md

+## Non-Goals
+
+* Add multi-tenancy to Thanos Ruler. 
+* Implement cross-tenant querying. It poses the question about how to track metrics of a multi-tenant query, probably requiring new logic for query splitting and/or separation of the concepts of query initiator tenant and query target tenants. To exemplify: if tenant A executes a query touching their own data and tenant B data (imagine a label matcher `tenant_id=~"A|B"`), how does Thanos export a request duration metric about this query? This is out of scope and an specific proposal can be created for it.


Interesting. I would argue we might need to be careful with adding feature that will block important features like cross tenancy in future. However I feel we get into the point where we talk about different tenancy aspects:

One is tenancy of data, so who owns what. Second is tenancy of query, who should be billed for query. Those write/read tenancy does not need to be 1:1. For example cross tenancy to me means that tenant A can read data of tenant B, but the quotas, settings and billing should be on behalf of one tenant.

All of this to me means we could refine this cross tenancy point. I think cross-tenancy is a goal, just we don't want shared tenancy for reads (?)

Cross-tenancy is definitely a future goal of the project and a good "follow up", but I didn't plan this proposal to go deep into its details. I don't think it'll block in any way a cross-tenant query implementation. Cross-tenant queries will require changes to the logic behind the label enforcement part, but there's no architectural decision that will need to be undone when the time comes.

Totally we will need to develop more on these two aspects of tenancy you mention, @bwplotka. Plus we will need some configuration that will work kinda like an authorization for cross-tenant query (i.e. tenant A can query B and C; tenant B can only query C, etc).

bwplotka · 2023-05-03T16:45:12Z

docs/proposals-accepted/202304-query-path-tenancy.md

+
+## How
+
+* Implement a mechanism to allow incoming requests to specify the tenant being queried using an HTTP header. This applies to all components in the query path. We follow the example of Thanos Receive, which uses the `--receive.tenant-header="THANOS-TENANT"` flag to configure the tenant header, adapting its name to each component. So in the Querier, the flag name will be `--querier.tenant-header`, in the Store Gateway it'll be `--store.tenant-header`, and so on.


Are we sure the tenant will be always represented by a single label not multiple labels?

Yup, our behavior should match the Receive's, which is using a single and configurable label, named tenant_id by default.

bwplotka · 2023-05-03T16:49:12Z

docs/proposals-accepted/202304-query-path-tenancy.md

+## How
+
+* Implement a mechanism to allow incoming requests to specify the tenant being queried using an HTTP header. This applies to all components in the query path. We follow the example of Thanos Receive, which uses the `--receive.tenant-header="THANOS-TENANT"` flag to configure the tenant header, adapting its name to each component. So in the Querier, the flag name will be `--querier.tenant-header`, in the Store Gateway it'll be `--store.tenant-header`, and so on.
+* The tenant header value from a given request should travel downstream to all the components being called, so that it can be added to their metrics, traces, and logs without requiring duplicated/extra work to re-parse the query. This also applies to gRPC calls.


To be strict for gRPC - will we pass that tenant as metadata or HTTP2 header? I think the metadata is the answer.

Agreed, @bwplotka. GRPC metadata seems to be the answer for it. Will batch this into updates to the proposal for the next push.

bwplotka · 2023-05-03T16:52:24Z

docs/proposals-accepted/202304-query-path-tenancy.md

+   2. Implement the `--querier.tenant-label-name` flag. If it is not empty, verify and enforce the tenant label in the query before it is handed to the query engine. As previously mentioned by the proposal, use prom-label-proxy's [Enforce.EnforceMatchers](https://github.com/prometheus-community/prom-label-proxy/blob/main/injectproxy/enforce.go#L141).
+   3. Update the metrics exported by the Querier to include a tenant label with the tenant indicated by the header.
+3. Store Gateway
+   1. Implement the `--store.tenant-header` (default is `""`") and `--query-frontend.default-tenant` (default is `"default-tenant"`) flags. Use the tenant header to identify the tenant in the metrics exported by the Store Gateway.


Also be careful as gRPC does not expose HTTP2 headers - metadata and trailers can be used instead. So flag name for gRPC would need to be different.

bwplotka · 2023-05-03T17:24:20Z

docs/proposals-accepted/202304-query-path-tenancy.md

+## Action Plan
+
+1. Query Frontend
+   1. Implement the `--query-frontend.tenant-header` (default is `""`") and `--query-frontend.default-tenant` (default is `"default-tenant"`) flags. Forward the tenant header to all the configured downstream query endpoints, if it is present.


Cool discussion, but I feel we all miscommunicated here a bit - some answers were a bit too repetitive and signalling some frustration. Let's calm down a bit, we all want to have the best solution here 🙈

My suggestions about the essence (let me know if I understood all right):

I think we tend to agree on the top most component (Querier, frontend or both) using header not request field. I would vote however to NOT add any flag to configure this unless community will need that to adopt it. Let's just agree on some hardcoded, opinionated header name for now and go from there.

Header vs Fields on downstream components and APIs. I think @fpetkovski raised a valid concern about too many flags. One solution is to move information to request field e.g StoreAPI. I would argue the better solution is to simply stick to one opinionated header name (hardcoded) for all components. If in unlikely case somebody will need a flag for it, we can discuss later (we would rather not).

To truly uncover what's best for no 2, I would unpack this item in Alternatives section, which actually @douglascamata wrote down (request vs header). I agree with @douglascamata to stay with headers, but I don't feel the arguments you used @douglascamata are convincing (: Maybe this is why we have some communication issues here. Let's unpack your points in the proposal:

This isn't the preferred solution because the Store API isn't used all the way from the topmost components (e.g. Querier and Query Frontend) to the bottommost one (e.g. Store Gateway). The Query Frontend talks to the Querier via the Query API. A Querier can talk to other Queriers using the Query API via gRPC. Finally, a Querier talks to Store Gateways using the Store API. All these APIs would have to be modified and the transmission of this information would differ depending on the component and API being used.

Fair, but also all components would need to agree on the header name that is very loosely defined and implemented so I feel this is neither improvement or disadvantage of a header (a tie )

In addition to this complication, this implementation does not enable any capability that can't be achieved with HTTP header approach.

Is it really a good argument? The opposite is also true - you can add field to all requests.

Essentially I feel we need different argumentation here:

Reference to Cortex (Pro header)

What semantically it makes sense more? (Pro header IMO as tenant belongs more to auth domain really)

Header consistency across components: Better code reusability and user understanding (Pro header)

Header consistency across HTTP APIs and gRPC Services. We don't have just StoreAPIs, we have other APIs and calls. Instead of extending all request messages we can just use single, consistent header (HTTP) and metadata item (gRPC).

Typed (Pro request)

docs/proposals-accepted/202304-query-path-tenancy.md

GiedriusS · 2023-05-04T07:39:56Z

docs/proposals-accepted/202304-query-path-tenancy.md

+## Non-Goals
+
+* Add multi-tenancy to Thanos Ruler. 
+* Implement cross-tenant querying. It poses the question about how to track metrics of a multi-tenant query, probably requiring new logic for query splitting and/or separation of the concepts of query initiator tenant and query target tenants. To exemplify: if tenant A executes a query touching their own data and tenant B data (imagine a label matcher `tenant_id=~"A|B"`), how does Thanos export a request duration metric about this query? This is out of scope and an specific proposal can be created for it.


So, it won't be possible to query data from multiple tenants anymore? 😢

@GiedriusS: it will still be possible, by default, to keep backwards compatibility. When the tenancy-enforcement feature is turned on on a Querier, then it won't be possible for requests arriving there. But you can have another set of Queriers with the feature disabled and multi-tenant queries will work, just like today. o/

@GiedriusS: I am trying to ensure we keep backwards compatibility and the possibility of running queries with and without tenant-enforcement in a cluster. Maybe we can collaborate later on a proposal for multi-tenant query with the tenancy enforcement on? It's not a goal of this proposal to outline how this would work. What do you think? :D

Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

douglascamata · 2023-05-05T16:55:29Z

@bwplotka @fpetkovski @saswatamcode: I believe I have handled and/or replied to all of your comments. Please do check the replies and the proposal in its current state when you have some time, we (me and the proposal) ready for a next round of feedback. :)

Thanks for your reviews 🙇

douglascamata · 2023-05-17T13:22:07Z

Could reviewers please give this a second look? Thank you for your time.

saswatamcode

Sorry for the delay on this! 🙏🏻

I think this proposal is in a good state, and this is a feature that we definitely want & need!

The proposal already covers a wide range of features across query path components, both for admin and end-user personas,

Per tenant o11y & billing (admin)
PromQL query tenancy (end-user)
Query limits (end-user & admin)

As the devil is in the details, it'd be good to keep in mind that while implementing, there might be certain limitations that haven't been considered, due to the breadth of this proposal. So being flexible, and updating this proposal at that point would be a great way to move forward! 🙂

Approving as I'd like to unblock and move this stream of work forward!

fpetkovski

Looks good to me as well, we can polish out the details during the implementation phase.

* Add 1st version of query path tenancy proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update proposal after initial feedback Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add cool picture Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Include example in cross tenant query complications Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve reasoning for why not using the QFE Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve writing in "How" section Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix owner profile link Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address few more PR review comments Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address feedback on flag name text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve non-goals text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update docs/proposals-accepted/202304-query-path-tenancy.md Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify scenario for pitfalls of current solution Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify that Store doesn't care about tenant label Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add an action plan Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention alternative idea of modifying Store API Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address lots of feedback on the proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Format query path tenancy proposal doc Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add a "Tenancy Model" subsection to "Goals" Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention header semanthics in comparison with gRPC message field Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve action plan structure and writing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com>

* Updates busybox SHA (#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (#6287) * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 Signed-off-by: Alex Le <leqiyue@amazon.com> * Reverted test code Signed-off-by: Alex Le <leqiyue@amazon.com> * Updated comment Signed-off-by: Alex Le <leqiyue@amazon.com> * docs: mismatch in changelog Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * Updates busybox SHA (#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com> Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: GitHub <noreply@github.com> Co-authored-by: Etienne Martel <etienne.martel.7@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Add CarTrade Tech as new adopter Signed-off-by: naveadkazi <navead@carwale.com> * sort ProxyResponseHeap comparing only diff labels Signed-off-by: Walther Lee <walthere.lee@gmail.com> * proxy heap: remove store labels before comparing response series Signed-off-by: Walther Lee <walthere.lee@gmail.com> * tests: Remove custom Between test matcher (#6310) * Remove custom Between test matcher The upstream PR to efficientgo/e2e has been merged, so we can use it from there. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove testingRespSet and update comparison func Signed-off-by: Walther Lee <walthere.lee@gmail.com> * Fix tests Signed-off-by: Walther Lee <walthere.lee@gmail.com> * query frontend, query UI: Native histogram support (#6071) * Implemented native histogram support for qfe and query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed marshalling for histograms in qfe Started working on native histogram query ui Copied histogram implementation for graph Added query range support for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Fixed sample and sample stream marshal fn Extended qfe native histogram e2e tests Added copyright to qfe queryrange compat Added query range test fo histograms and try to fix ui tests Fixed DataTable test Review feedback Fixed native histogram e2e test Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add histogram support for ApplyCounterResetsSeriesIterator Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add chnagelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed PrometheusResponse minTime for histograms in qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated prometheus common to v0.40.0 and queryrange.Sample fixes Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated Readme Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Addressed PR comments Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> trigger tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed tsdbutil references Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Enabled pushdown for query native hist test and removed ToDo Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Refactored native histogram query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * store: add streamed snappy encoding for postings list (#6303) * store: add streamed snappy encoding for postings list We've noticed that decoding Snappy compressed postings list takes a lot of RAM: ``` (pprof) top Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total Dropped 1069 nodes (cum <= 10.56GB) Showing top 10 nodes out of 82 flat flat% sum% cum cum% 0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1 2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings 1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings 2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings 1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode 1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode ``` This is because we are creating a new []byte slice for the decoded data each time. To avoid this RAM usage problem, let's stream the decoding from a given buffer. Since Snappy block format doesn't support streamed decoding, let's switch to Snappy stream format which is made for exactly that. Notice that our current `index.Postings` list does not support going back through Seek() even if theoretically one could want something like that. Fortunately, to search for posting intersection, we need to only go forward. Benchmark data: ``` name time/op PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3% PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4% PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1% PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4% PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1% PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13% PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1% PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4% PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1% PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9% PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3% PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4% name alloc/op PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0% PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0% name allocs/op PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0% ``` Compression ratios are still the same like previously: ``` $ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store [snip] === RUN TestDiffVarintCodec/snappy/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44498 bytes postings_codec_test.go:81: ratio: 0.012 === RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44670 bytes postings_codec_test.go:81: ratio: 0.012 ``` Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: clean up postings code Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: fix estimation Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: use buffer.Bytes() Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store/postings_codec: reuse extgrpc compressors/decompressors Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: add item Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: clean up whitespace Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * compact: atomically replace no compact marked map (#6319) With lots of blocks it could take some time to fill this no compact marked map hence replace it atomically. I believe this leads to problems in the compaction planner where it picks up no compact marked blocks because meta syncer does synchronizations concurrently. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Reuse buffers for label comparison Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use len instead of cap Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use len instead of cap Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fixed Jaeger exporter specification link (#6324) Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * receive: Make max backoff configurable (#6163) Signed-off-by: Philip Gough <philip.p.gough@gmail.com> * Replace summary in extprom metrics with histogram (#6327) * Replaced summary in extprom metrics with histogram Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused parameters from NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Reverted NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Expose info for each TSDB This commit exposes the label set alongside the min and max time for each TSDB covered by a Store. This information is used to scope the min time for a remote query so that we do not produce partial aggregates in distriuted mode. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add test case for proxy store Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Pass engine factory by reference Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Cleanup Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Use NewQueryEngineFactory in tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Avoid expensive log.Valuer evaluation for disallowed levels (#6322) Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Fix inconsistent error for series limits in Store API (#6330) * store: fix inconsistent error for series limits Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * rename labelValues serires liimiter test function Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * *: remove unmaintained gzip library (#6332) Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Traces sampler env var (#6306) * Issue#5947 OTEL_TRACES_SAMPLER env var Signed-off-by: shayyxi <shazi12384@gmail.com> * Test correction Signed-off-by: shayyxi <shazi12384@gmail.com> * doc failure correction. parse float argument correction. Signed-off-by: shayyxi <shazi12384@gmail.com> * added the changelog. Signed-off-by: shayyxi <shazi12384@gmail.com> * ran make docs to fix the build failure. Signed-off-by: shayyxi <shazi12384@gmail.com> * corrected the incorrect change in tools.md Signed-off-by: shayyxi <shazi12384@gmail.com> * fixed review comments. Signed-off-by: shayyxi <shazi12384@gmail.com> --------- Signed-off-by: shayyxi <shazi12384@gmail.com> Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com> Co-authored-by: shayyxi <shazi12384@gmail.com> * query: use storepb.SeriesServer (#6334) Use storepb.SeriesServer instead of the concrete struct. This allows implementing functionality on top of the proxy. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. https://github.com/redis/rueidis/issues/209 (#6342) * use github.com/onsi/gomega/gleak to detect goroutine leak with timeout Signed-off-by: Rueian <rueiancsie@gmail.com> * fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (#6345) Signed-off-by: aimuz <mr.imuz@gmail.com> * Return grpc code resource exhausted for byte limit error (#6325) * return grpc code resource exhausted for byte limit error Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * update partial response strategy Signed-off-by: Ben Ye <benye@amazon.com> * fix limit Signed-off-by: Ben Ye <benye@amazon.com> * try to fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test error message Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Bump promql-engine to fix https://github.com/thanos-community/promql-engine/issues/239 (#6349) Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> * Optimize generating store match reason (#6355) Removes reason generation when debug logging is not enabled. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Updates busybox SHA (#6365) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Query: Add +Inf bucket to query duration metrics (#6358) * Query: Add +Inf bucket to query duration metrics For the query duration metrics (`thanos_store_api_query_duration_seconds`), we record query respond latency, based on the size of the query (samples/series), and save to a histogram. However, when a query is made which exceeds the biggest sample/serie size, we would prior to this commit, put the request into the largest bucket. With this commit, we instead create an `+Inf` bucket, and put requests which are larger than the biggest defined bucket into that. This gives more accurate results, and also allow one to see if the bucket sizes are incorrectly sized. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Tests: Mutex around non-thread safe random source When creating test blocks, we use a non-thread safe random source, in multiple goroutines. Due to this, tests would sometime panic. This commits puts a mutex around calls using the same source, in order to avoid this. This should hopefully improve reliability of e2e tests. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Check context when expanding postings (#6363) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * ui: only keep name in store_matches param (#6371) We are doing store matching on the `name` field hence only keep that field in the URL because otherwise the URL could get quite lengthy with external labelsets inside of it. Besides unit tests, I have also tested locally: - Enable store filtering; - Select store(-s); - Copy/paste URL into the new tab and see that the same stores are loaded like expected; - See that URL only has names in them. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * docs: replace --store with --endpoint Replace deprecated `--store` with `--endpoint` in docs. Signed-off-by: Paul Gier <paul.gier@datastax.com> * Optimizing "grafana generated" regex matchers (#6376) * Opmizing Group Regex Signed-off-by: Alan Protasio <alanprot@gmail.com> * fixing native histogram tests Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Cache: various index cache client improvements (#6374) * e2e(query): Reproduce dedup issue from #6257 Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add dedup e2e test for Receive With internal and external labels support. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Simplify generated blocks for query test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve query dedup test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Write a query test for dedup with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Refactor query dedup test with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive query test Now it properly ensures the double dedup works (on internal and external labels). Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix receive drawing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add one extra test caes for query dedup from store Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Receive query with dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Sidecar query dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Query Explanation (#6346) * Return Query Explaination in QueryAPI A param `explain` is added to QueryAPI, if true then explanation returned by the `Explain()` method of the query having structure `ExplainOutputNode` is returned in response. Query Explanation is added under new field in response that is `thanosInfo`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add explain checkbox in thanos UI A explain checkbox is added to Thanos Query UI, that requests for query explanation from thanos query api. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add ExpandableNode Component ExpandableNode component renders Query Explanation in the thanos UI. Requires a new package `react-accessible-treeview`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Disable Explain checkbox on prometheus engine Prometheus engine sends out error if toggle explain button. To provide better experience, the explain checkbox get disbaled on switching to prometheus engine and enable back on switching to thanos engine. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add alert box with horizontal scrolling for Explanation Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove ExpandableNode and Add ListTree Updates the design for query explanation box, removes `ExpandableNode` and the dependency. Builts a new `ListTree` that does the same using reactstrap and custom css. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor refactor in Query API response `thanosInfo` is removed from Query reponse and used `explanation` directly. `disableCheckbox` is also renamed to `disableExplainCheckbox` in thanos UI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI tests to passing Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor UI changes and test fix UI improvements and Panel test fix other way around, resetting the results on panel construction. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update promql-engine to use Explain method Signed-off-by: Pradyumna Krishna <git@onpy.in> * Build UI assets Build UI assets, that runs new thanos UI with explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Revert proxy url change from package.json `proxy` was accidently changed and committed with package.json when removed dependency. Hence, reverting it back. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor changes in UI Fix requested changes in UI. - Rename `state` and `setState` to `mapping` and `setMapping`. - Rename `NodeTree` to `QueryTree`. - Use unicode characters instead of `-` and `+`. - Fix blue box on explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI assets Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (#6379) * Implementing Regex optimization on the MatchNotRegexp matcher type Signed-off-by: Alan Protasio <alanprot@gmail.com> * Opmizing MatchNotEqual Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Expected failure of block label query dedup tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Put back the correct makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * add de-cix as adopter (#6386) Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de> * Remove extra line that broke untouched test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add back line break at end of makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive single ingestor test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup issue in Receive Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add even more test cases for dedup on store gw Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup bug in Sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reuse nginx image name Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Let all users read the metrics file from static metrics server Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * [chore] Updating Query Engine and Prometheus (#6392) * Updating Query Engine Signed-off-by: Alan Protasio <alanprot@gmail.com> * fix prometheus breaking change Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update prometheus with https://github.com/prometheus/prometheus/pull/12387 Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Receive: Allow specifying tenant-specific external labels in RouterIngestor (#5777) Signed-off-by: haanhvu <haanh6594@gmail.com> * check context cancel when doing posting batches (#6396) Signed-off-by: Ben Ye <benye@amazon.com> * Expose store gateway query stats in series response hints (#6352) * expose query stats hints Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * add query stats hints in result Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * add merge method Signed-off-by: Ben Ye <benye@amazon.com> * fix unit test Signed-off-by: Ben Ye <benye@amazon.com> modify hints proto Signed-off-by: Ben Ye <benye@amazon.com> fix unit test Signed-off-by: Ben Ye <benye@amazon.com> update format Signed-off-by: Ben Ye <benye@amazon.com> * update comments Signed-off-by: Ben Ye <benye@amazon.com> * try again Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * receive: make az aware ketama hashring (#6369) * receive: make az aware ketama hashring Signed-off-by: Alexander Rickardsson <alxric@aiven.io> * receive: pass endpoints in hashring config as object Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive: add some tests for consistent hashing in presence of AZs Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive,docs: add migration note for az aware hashring Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> --------- Signed-off-by: Alexander Rickardsson <alxric@aiven.io> Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Reformat asciiflow chart Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Proposal: query path tenancy (#6320) * Add 1st version of query path tenancy proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update proposal after initial feedback Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add cool picture Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Include example in cross tenant query complications Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve reasoning for why not using the QFE Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve writing in "How" section Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix owner profile link Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address few more PR review comments Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address feedback on flag name text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve non-goals text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update docs/proposals-accepted/202304-query-path-tenancy.md Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify scenario for pitfalls of current solution Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify that Store doesn't care about tenant label Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add an action plan Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention alternative idea of modifying Store API Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address lots of feedback on the proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Format query path tenancy proposal doc Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add a "Tenancy Model" subsection to "Goals" Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention header semanthics in comparison with gRPC message field Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve action plan structure and writing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Fix double-counting bug in http_request_duration metric (#6399) * fix double-counting bug in http_request_duration metric Signed-off-by: 4orty <kwk5178@gmail.com> * Update Changelog Signed-off-by: 4orty <kwk5178@gmail.com> --------- Signed-off-by: 4orty <kwk5178@gmail.com> * Replace group with resolution in compact metrics. Compaction metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a `resolution` label to the compaction counters. Fixes: https://github.com/thanos-io/thanos/issues/5841 Signed-off-by: SuperQ <superq@gmail.com> * Reuse static metrics server from e2e framework Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Updates busybox SHA (#6403) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Fix series stats merge (#6408) * fix series stats merge Signed-off-by: Ben Ye <benye@amazon.com> * update license header Signed-off-by: Ben Ye <benye@amazon.com> * use reflect Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: allow unlimited head_series_limit tenants (#6406) With this commit we now allow to configure tenants with unlimited active series limit by setting the limit to `0`. Prior to this commit setting a per tenant limit to `0` would cause the tenant to be unable to write any metrics at all. This fixes: https://github.com/thanos-io/thanos/issues/6393 Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * expose downloaded data size in query hints (#6409) Signed-off-by: Ben Ye <benye@amazon.com> * maintainers: add myself to triagers (#6414) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Add `@douglascamata` to triagers (#6418) Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add Blog (#6411) * Add LFX blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add Headers to blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Lint blog Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * blog: Fix images for LFX post (#6422) * blog: Fix images for LFX post Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * fix lint Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Index Cache: Change cache key for postings (#6405) * extend postings cache key with codec Signed-off-by: Ben Ye <benye@amazon.com> * add changelog Signed-off-by: Ben Ye <benye@amazon.com> * update code back Signed-off-by: Ben Ye <benye@amazon.com> * add colon Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix another test Signed-off-by: Ben Ye <benye@amazon.com> * add compression scheme const to remote index cache Signed-off-by: Ben Ye <benye@amazon.com> * address required comments Signed-off-by: Ben Ye <benye@amazon.com> * fix compression scheme name Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: upgrading logs for failed uploads to error (#6427) * FIX: upgrading log for failed upload to error Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * docs: added changelog entry Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> --------- Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * fix postings test Signed-off-by: Ben Ye <benye@amazon.com> * Add aiven as adopter... more soon! Signed-off-by: Jonah Kowall <jkowall@kowall.net> * First draft of the Aiven adoption story Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add aiven as adopter... more soon! (#6430) Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Report gRPC connnection errors to the caller (#6428) By default `grpc.DialContext()` is non-blocking so any connection issue will not be surfaced to the user. This change makes it blocking and configures the gRPC dialer to report the underlying error if any happens. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * chore: remove duplicated `gopkg.in/fsnotify.v1` dep (#6432) * chore: remove duplicated `gopkg.in/fsnotify.v1` dep `github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same dependency. We can keep `github.com/fsnotify/fsnotify` and remove `gopkg.in/fsnotify.v1`. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * docs: add changelog Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> --------- Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * Expose estimated chunk and series size as configurable options (#6426) * expose estimated chunk and series size as configurable options Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: make tsdb stats limit configurable (#6437) * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> --------- Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * *: wire new Engine/Explain fields in query-frontend (#6433) - Pass Engine/Explain fields in query-frontend codecs - Add Engine field to QFE cache key - Add e2e tests for all cases Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index cache: Cache expanded postings (#6420) * cache expanded postings in index cache Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * rebase main and added compression name to key Signed-off-by: Ben Ye <benye@amazon.com> * update key Signed-off-by: Ben Ye <benye@amazon.com> * add e2e test for memcached Signed-off-by: Ben Ye <benye@amazon.com> * fix cache config Signed-off-by: Ben Ye <benye@amazon.com> * address review comments Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * add approximate series size to index stats (#6425) Signed-off-by: Ben Ye <benye@amazon.com> * index stats: fix chunk size calculation (#6424) Signed-off-by: Ben Ye <benye@amazon.com> * Remove some unused Cortex vendored code and metrics (#6440) * Avoid direct cortex vendor dependency in promclient (#6443) * use own explanation struct Signed-off-by: Ben Ye <benye@amazon.com> * omit empty Signed-off-by: Ben Ye <benye@amazon.com> * fix e2e test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * query.md: Added grpc_{client,server}_handled_total to metrics list. (#6351) Fixes: #6280 Signed-off-by: Jonathan Davies <jpds@protonmail.com> * Updates busybox SHA (#6448) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * store: read postings directly into delta encoded format (#6442) Instead of allocating bytes for raw postings, let's read them directly into diff varint format to save memory. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * compact: Add Index Stats to block metadata (#6441) * compactor set index stats on block Signed-off-by: Ben Ye <benye@amazon.com> * comment Signed-off-by: Ben Ye <benye@amazon.com> * change field Signed-off-by: Ben Ye <benye@amazon.com> * update downsample Signed-off-by: Ben Ye <benye@amazon.com> * add index stats into compaction Signed-off-by: Ben Ye <benye@amazon.com> * fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * estimate block chunk and series size from metadata (#6449) Signed-off-by: Ben Ye <benye@amazon.com> * Adding graphic for blog showing architecture Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing mermaid graph Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing mermaid graph again Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Remove image and fix mermaid for other flowchart Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing prod flowchart Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing prod flowchart title Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing md table Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing md table again Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Store: fix crash on empty regex matcher Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Some corrections on text and links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing diagams Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing currencies Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing diagams Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing images Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Website: Fix blog images width (#6450) * Set blog post images width to 100% Blog Post images width isn't set and hence the images doesn't get fit to the page. Setting width to 100% make sure images fit to the web page and screen size. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove width constraint from blog images Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Cache calculated mint and maxt for each remote engine (#6458) The distributed optimizer in the Thanos engine calls MinT() and LabelSets multiple times on each remote engine. These calculations can get expensive when an engine covers a large amount of TSDBs. This commit introduces caching of the calculated values for a remote engine's mint, maxt and label sets. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cache empty expanded postings (#6464) Signed-off-by: Ben Ye <benye@amazon.com> * mixin/receive: add limits alerting (#6466) * add some thanos receive limits alerts Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix and generate examples Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * quick fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * quick fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix test Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix invalid changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update docs/blog/2023-06-08-thanos-at-aiven.md Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add retention for M3 Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add image versus mermaid diagram. Couldn't comment it out unfortunately Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding text around link to fix linter Signed-off-by: Jonah Kowall <jkowall@kowall.net> * make docs Signed-off-by: Jonah Kowall <jkowall@kowall.net> * add alert for tenant reaching head series limit (#6467) Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * go.mod: bump rest of otel libs (#6447) * go.mod: bump rest of otel libs Bump these two OTEL libs to be like the rest of the OTEL libs. We had a strange issue where `service.name` was not set even though it was in environment variables. Bumping these libs fixes the problem. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * otlp: only set service.name if provided Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * e2e: add test case Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: disable pooling for postings benchmarks (#6473) Pooling hides the real allocations in postings benchmarks because the allocation only happens once really. In reality this is different when multiple readers are created at once. Disable pooling in benchmarks to understand what is happening. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Fixing date and author for Aiven blog (#6476) * Fixing authors Signed-off-by: Jonah Kowall <jkowall@kowall.net> * make docs and fixing date Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing authors Signed-off-by: Jonah Kowall <jkowall@kowall.net> --------- Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Check context when expanding postings 2nd attempt (#6471) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * import Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * [FEAT] Add thanos ruler blog post (#6477) Signed-off-by: Hélia Barroso <helia.barroso@farfetch.com> * blog: Small fixes to Aiven article (#6481) * Correct some links and typos in 2023-06-08-thanos-at-aiven.md The linkified URLs in the article were including the closing bracket, breaking the links. Make them proper Markdown links. Also fixed a few other typos I spotted. Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> * Update 2023-06-08-thanos-at-aiven.md Clarify a couple of things in the final paragraph. Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> --------- Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> * Rename series and postings fetch duration metrics (#6479) * fix wrong metric names for series and postings fetch duration Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * note breaking change Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Update to prometheus/common v0.44.0 (#6483) Signed-off-by: Dan Bason <colonel.dan@gmail.com> * go.mod: update rueidis client (#6485) The latest improvements have great perf improvements. Let's update it. Thank you to [rueian](https://github.com/rueian) and all of the contributors to that awesome Redis client! Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Introduce tenancy package (#6482) This commit introduces a tenancy package in preparation for introduction of tenancy in the query path. Tenancy related code which can be common between components are moved out of the receive component into the tenancy package. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Ignore nginx.com links in mdox (#6489) Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * blog: Update Aiven blog post (#6487) Wording changes and better formatting for the tables. Signed-off-by: Simon Brüggen <simon.bruggen@aiven.io> * store: optimized snappy streamed reading (#6475) Optimize snappy streamed reading by traversing through the byte slice in advance to determine how big of a buffer we will need. I have opted to rewrite snappy streamed format decoding because it is a straightforward format. Complex parts were deferred to klauspost/compress. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Remove Exists call in meta fetcher (#6474) * Remove Exists call in meta fetcher The meta fetcher lists the bucket by iterating through top level paths from the entire keyspace, and then for each key calls Exists on the meta.json file. This leads to an N+1 amplification against object storage, manifesting as a very high increase in costs and occasional throttling by cloud providers. This commit changes the iteration logic to recursively list all files instead, which can be done with far fewer API calls to object storage. Meta files are then identified from the keys returned by the list operation instead of making one API request from each file. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Revert e2e test change Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * e2e/store: use now instead of time.Now() each time (#6493) Consistently use the same timestamp instead of the current time each time during a test. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * [FIX] Documentation issue (#6494) Signed-off-by: Hélia Barroso <helia_barroso@hotmail.com> * Updates busybox SHA (#6497) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * chore: pkg imported more than once (#6499) * chore: pkg imported more than once Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> * fix: remove pkg alias Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> --------- Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> * *: Remove unnecessary configuration reload from `ContentPathReloader` and improve its tests (#6496) * Fix and improve PathContentReloader tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run tests in parallel Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add changelog entry Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix table tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> * add context check when decoding cached postings (#6506) Signed-off-by: Ben Ye <benye@amazon.com> * *: Change the engine behind `ContentPathReloader` to be completely independent of any filesystem concept. (#6503) * Add fallback config reload for symlinks Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve pooling config reload Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make tests more reliable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve tests again Assert on runutil.Repeat output Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Pass the debounce/reload time to each test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add comments and ensure interfaces are implemented Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make the limit configuration reload time customizable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * goimports file Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix lint warning Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve log for polling engine Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make linter happy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Extract symlink identification to a separate function Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix links to go-grpc-middleware after v2 merge into main Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add period to make linter happy. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog entry Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Replace the fsnotify based engine from `PathContentReloader` with the polling one Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove check for empty filePath Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Unexpose pollingEngine.Start Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix pollingEngine doc-comment Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Log file path when config is reloaded Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * receive: add float histogram support (#6323) * Added receive float histogram support Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added comments for fns copied from Prometheus Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Improved comment Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unnecessary if Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed native histogram proto conversion in remote engine Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused histogram conversion from remote engine Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fix and renaming in native_histograms_test.go Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Trigger Build Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * go.mod: bump e2e framework version (#6516) Bump e2e framework version and add a note about E2E_DOCKER_CPUS. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Add changelog for #6474 (#6518) Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cortex/redisclient: use rueidis client (#6520) * cortex/redisclient: use rueidis client Use the same rueidis client in query-frontend. Solves https://github.com/thanos-io/thanos/issues/6094. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cortex/cache: gofumpt + fix errors.Errorf Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil/docs: clean up more old stuff Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index header: Remove memWriter from fileWriter (#6509) * index header: remove memWriter from fileWriter Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * refactor Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * update comment Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Wrap object store Prometheus registry (#6152) * Wrap object store Prometheus registery In preparation for the work being done at https://github.com/thanos-io/objstore/pull/26. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update objstore to the latest version This version removes the `thanos_` prefix from metrics, which is the reason for wrapping the objstore's metrics registry in the first place. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix modules Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Wrap metrics registerer for objstore bucket Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove prefix from Thanos Store metrics Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix goimports Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Put back upgraded objstore dep Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Move to more recent ref of thanos/objstore Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Ignore OBS in objstore tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix linting error on main test after objstore upgrade Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Skip OCS objstore test in circle ci Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix echo in makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * e2e/store: try to fix Series() limit test again (#6522) I finally managed to reproduce this failure locally with https://github.com/efficientgo/e2e/commit/c316eb95ae5b0fa65b0be110f8cfc3c48e8810bb. The added t.Logf() showed that is the problem is that with a lower bytes limit, it might hit the series or chunks part first. I have bumped the bytes limit. I calculated the new bytes limit by checking how much bytes are allocated before sending the last chunk. I have also noticed that one block is created without a delay. Update it so that it would be like the others. Include objstore@main update with https://github.com/thanos-io/objstore/pull/62/files so that Iter() would always return an error on a timeout. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * pkg/reloader: use watchInterval timeout for initial apply (#6519) * pkg/reloader: use watchInterval timeout for initial apply Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com> * changelog Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com> * use distinct context for initial sync Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com> --------- Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com> * Standardize index cache metrics (#6523) * Add histogram metrics for index cache item size (#6528) * add histogram metrics for index cache item size Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Upgrade objstore (#6507) * Upgrade objstore Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Update with the latest APIs Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> --------- Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> * Make compact lifecycle more flexible to be overridden for sharded compaction (#5964) * Updates busybox SHA (#6534) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * BucketedBytes to buffer byte slices when decoding postings from cache (#6531) * BucketedBytes to buffer byte slices when deconding postings from cache Signed-off-by: Alan Protasio <alanprot@gmail.com> * Fixing Lint Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Shipper: change upload compacted type from bool to a function (#6526) * change shipper upload compacted type from bool to a function Signed-off-by: Ben Ye <benye@amazon.com> * add default to false Signed-off-by: Ben Ye <benye@amazon.com> * reset uploadedCompacted to 0 Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Query: Forward tenant information via StoreAPI (#6530) * Querier: Forward tenant information downstream With this commit we attach tenant information to each query request and forward it via the StoreAPI to any downstream Store Gateways and Queriers. We add the following command lines options which mimics the tenant functionality in Receive. The options are currently hidden, as they provide no real functionality yet. This will come in future steps. --query.tenant-header --query.default-tenant --query.tenant-certificate Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Receive: Use CertificateField from Tenancy pkg These consts are now defined in the Tenancy package, so we should use those instead. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Deduplicate matchers in posting group (#6532) * deduplicate matchers in posting group Signed-off-by: Ben Ye <benye@amazon.com> * don't use map for deduplication Signed-off-by: Ben Ye <benye@amazon.com> * lint Signed-off-by: B…

* fix 0.31 changelog (#6278) Signed-off-by: junot <junotxiang@kubesphere.io> * Query: Switch Multiple Engines (#6234) * Query: Switch engines using `engine` param Thanos query has two engine, prometheus (default) and thanos. A single engine runs through thanos query command at a time, and have to re run the command to switch between. This commit adds a functionality to run multiple engines at once and switch between them using `engine` query param inq query api. To avoid duplicate matrics registration, the thanos engine is provided with a different registerer having prefix `tpe_` (not been finalized yet). promql-engine command line flag has been removed that specifies the query engine to run. Currently this functionality not implemented on GRPCAPI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add multiple engine support to GRPCAPI Fix build fail for thanos, adds support for multiple engine in GRPCAPI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Create QueryEngineFactory to create engines QueryEngineFactory makes a collection for all promql engines used by thanos and returns it. Any engine can be created and returned using `GetXEngine` method. It is currently limited to 2 engines prometheus and thanos engines that get created on the first call. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use QueryEngineFactory in query API thanos query commands pass `QueryEngineFactory` to query apis that will use engine based on query params. It will provide more flexibility to create multiple engines in thanos. Adds `defaultEngine` CLI flag, A default engine to use if not specified with query params. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update Query API tests Fixes breaking tests Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor changes and Docs fixes * Move defaultEngine argument to reduce diff. * Generated Docs. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add Engine Selector/ Dropdown to Query UI Engine Selector is a dropdown that sets an engine to be used to run the query. Currently two engines `thanos` and `prometheus`. This dropdown sends a query param `engine` to query api, which runs the api using the engine provided. Provided to run query using multiple query engines from Query UI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Move Engine Selector to Panel Removes Dropdown component, and renders Engine Selector directly. Receives defaultEngine from `flags` API. Updates parseOptions to parse engine query param and updates test for Panel and utils. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Upgrade promql-engine dependency Updates promql-engine that brings functionality to provide fallback engine using enigne Opts. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add MinT to remote client MinT method was missing from Client due to updated promql-engine. This commits adds mint to the remote client. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use prometheus fallback engine in thanos engine Thanos engine creates a fallback prometheus engine that conflicts with another prometheus engine created by thanos, while registering metrics. To fix this, provided created thanos engine as fallback engine to thanos engine in engine Opts. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use enum for EngineType in GRPC GRPC is used for communication between thanos components and defaultEngine was a string before. Enum makes more sense, and hence the request.Enigne type has been changed to querypb.EngineType. Default case is handled with another default value provided in the enum. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update query UI bindata.go Compile react app using `make assets`. Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * docs: mismatch in changelog Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> * Updates busybox SHA (#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (#6287) * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 Signed-off-by: Alex Le <leqiyue@amazon.com> * Reverted test code Signed-off-by: Alex Le <leqiyue@amazon.com> * Updated comment Signed-off-by: Alex Le <leqiyue@amazon.com> * docs: mismatch in changelog Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * Updates busybox SHA (#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com> Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: GitHub <noreply@github.com> Co-authored-by: Etienne Martel <etienne.martel.7@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Add CarTrade Tech as new adopter Signed-off-by: naveadkazi <navead@carwale.com> * sort ProxyResponseHeap comparing only diff labels Signed-off-by: Walther Lee <walthere.lee@gmail.com> * proxy heap: remove store labels before comparing response series Signed-off-by: Walther Lee <walthere.lee@gmail.com> * tests: Remove custom Between test matcher (#6310) * Remove custom Between test matcher The upstream PR to efficientgo/e2e has been merged, so we can use it from there. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove testingRespSet and update comparison func Signed-off-by: Walther Lee <walthere.lee@gmail.com> * Fix tests Signed-off-by: Walther Lee <walthere.lee@gmail.com> * query frontend, query UI: Native histogram support (#6071) * Implemented native histogram support for qfe and query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed marshalling for histograms in qfe Started working on native histogram query ui Copied histogram implementation for graph Added query range support for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Fixed sample and sample stream marshal fn Extended qfe native histogram e2e tests Added copyright to qfe queryrange compat Added query range test fo histograms and try to fix ui tests Fixed DataTable test Review feedback Fixed native histogram e2e test Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add histogram support for ApplyCounterResetsSeriesIterator Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add chnagelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed PrometheusResponse minTime for histograms in qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated prometheus common to v0.40.0 and queryrange.Sample fixes Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated Readme Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Addressed PR comments Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> trigger tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed tsdbutil references Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Enabled pushdown for query native hist test and removed ToDo Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Refactored native histogram query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * store: add streamed snappy encoding for postings list (#6303) * store: add streamed snappy encoding for postings list We've noticed that decoding Snappy compressed postings list takes a lot of RAM: ``` (pprof) top Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total Dropped 1069 nodes (cum <= 10.56GB) Showing top 10 nodes out of 82 flat flat% sum% cum cum% 0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1 2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings 1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings 2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings 1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode 1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode ``` This is because we are creating a new []byte slice for the decoded data each time. To avoid this RAM usage problem, let's stream the decoding from a given buffer. Since Snappy block format doesn't support streamed decoding, let's switch to Snappy stream format which is made for exactly that. Notice that our current `index.Postings` list does not support going back through Seek() even if theoretically one could want something like that. Fortunately, to search for posting intersection, we need to only go forward. Benchmark data: ``` name time/op PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3% PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4% PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1% PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4% PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1% PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13% PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1% PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4% PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1% PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9% PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3% PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4% name alloc/op PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0% PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0% name allocs/op PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0% ``` Compression ratios are still the same like previously: ``` $ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store [snip] === RUN TestDiffVarintCodec/snappy/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44498 bytes postings_codec_test.go:81: ratio: 0.012 === RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44670 bytes postings_codec_test.go:81: ratio: 0.012 ``` Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: clean up postings code Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: fix estimation Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: use buffer.Bytes() Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store/postings_codec: reuse extgrpc compressors/decompressors Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: add item Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: clean up whitespace Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * compact: atomically replace no compact marked map (#6319) With lots of blocks it could take some time to fill this no compact marked map hence replace it atomically. I believe this leads to problems in the compaction planner where it picks up no compact marked blocks because meta syncer does synchronizations concurrently. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Reuse buffers for label comparison Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use len instead of cap Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use len instead of cap Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fixed Jaeger exporter specification link (#6324) Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * receive: Make max backoff configurable (#6163) Signed-off-by: Philip Gough <philip.p.gough@gmail.com> * Replace summary in extprom metrics with histogram (#6327) * Replaced summary in extprom metrics with histogram Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused parameters from NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Reverted NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Expose info for each TSDB This commit exposes the label set alongside the min and max time for each TSDB covered by a Store. This information is used to scope the min time for a remote query so that we do not produce partial aggregates in distriuted mode. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add test case for proxy store Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Pass engine factory by reference Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Cleanup Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Use NewQueryEngineFactory in tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Avoid expensive log.Valuer evaluation for disallowed levels (#6322) Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Fix inconsistent error for series limits in Store API (#6330) * store: fix inconsistent error for series limits Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * rename labelValues serires liimiter test function Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * *: remove unmaintained gzip library (#6332) Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Traces sampler env var (#6306) * Issue#5947 OTEL_TRACES_SAMPLER env var Signed-off-by: shayyxi <shazi12384@gmail.com> * Test correction Signed-off-by: shayyxi <shazi12384@gmail.com> * doc failure correction. parse float argument correction. Signed-off-by: shayyxi <shazi12384@gmail.com> * added the changelog. Signed-off-by: shayyxi <shazi12384@gmail.com> * ran make docs to fix the build failure. Signed-off-by: shayyxi <shazi12384@gmail.com> * corrected the incorrect change in tools.md Signed-off-by: shayyxi <shazi12384@gmail.com> * fixed review comments. Signed-off-by: shayyxi <shazi12384@gmail.com> --------- Signed-off-by: shayyxi <shazi12384@gmail.com> Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com> Co-authored-by: shayyxi <shazi12384@gmail.com> * query: use storepb.SeriesServer (#6334) Use storepb.SeriesServer instead of the concrete struct. This allows implementing functionality on top of the proxy. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. https://github.com/redis/rueidis/issues/209 (#6342) * use github.com/onsi/gomega/gleak to detect goroutine leak with timeout Signed-off-by: Rueian <rueiancsie@gmail.com> * fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (#6345) Signed-off-by: aimuz <mr.imuz@gmail.com> * Return grpc code resource exhausted for byte limit error (#6325) * return grpc code resource exhausted for byte limit error Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * update partial response strategy Signed-off-by: Ben Ye <benye@amazon.com> * fix limit Signed-off-by: Ben Ye <benye@amazon.com> * try to fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test error message Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Bump promql-engine to fix https://github.com/thanos-community/promql-engine/issues/239 (#6349) Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> * Optimize generating store match reason (#6355) Removes reason generation when debug logging is not enabled. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Updates busybox SHA (#6365) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Query: Add +Inf bucket to query duration metrics (#6358) * Query: Add +Inf bucket to query duration metrics For the query duration metrics (`thanos_store_api_query_duration_seconds`), we record query respond latency, based on the size of the query (samples/series), and save to a histogram. However, when a query is made which exceeds the biggest sample/serie size, we would prior to this commit, put the request into the largest bucket. With this commit, we instead create an `+Inf` bucket, and put requests which are larger than the biggest defined bucket into that. This gives more accurate results, and also allow one to see if the bucket sizes are incorrectly sized. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Tests: Mutex around non-thread safe random source When creating test blocks, we use a non-thread safe random source, in multiple goroutines. Due to this, tests would sometime panic. This commits puts a mutex around calls using the same source, in order to avoid this. This should hopefully improve reliability of e2e tests. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Check context when expanding postings (#6363) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * ui: only keep name in store_matches param (#6371) We are doing store matching on the `name` field hence only keep that field in the URL because otherwise the URL could get quite lengthy with external labelsets inside of it. Besides unit tests, I have also tested locally: - Enable store filtering; - Select store(-s); - Copy/paste URL into the new tab and see that the same stores are loaded like expected; - See that URL only has names in them. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * docs: replace --store with --endpoint Replace deprecated `--store` with `--endpoint` in docs. Signed-off-by: Paul Gier <paul.gier@datastax.com> * Optimizing "grafana generated" regex matchers (#6376) * Opmizing Group Regex Signed-off-by: Alan Protasio <alanprot@gmail.com> * fixing native histogram tests Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Cache: various index cache client improvements (#6374) * e2e(query): Reproduce dedup issue from #6257 Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add dedup e2e test for Receive With internal and external labels support. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Simplify generated blocks for query test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve query dedup test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Write a query test for dedup with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Refactor query dedup test with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive query test Now it properly ensures the double dedup works (on internal and external labels). Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix receive drawing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add one extra test caes for query dedup from store Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Receive query with dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Sidecar query dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Query Explanation (#6346) * Return Query Explaination in QueryAPI A param `explain` is added to QueryAPI, if true then explanation returned by the `Explain()` method of the query having structure `ExplainOutputNode` is returned in response. Query Explanation is added under new field in response that is `thanosInfo`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add explain checkbox in thanos UI A explain checkbox is added to Thanos Query UI, that requests for query explanation from thanos query api. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add ExpandableNode Component ExpandableNode component renders Query Explanation in the thanos UI. Requires a new package `react-accessible-treeview`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Disable Explain checkbox on prometheus engine Prometheus engine sends out error if toggle explain button. To provide better experience, the explain checkbox get disbaled on switching to prometheus engine and enable back on switching to thanos engine. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add alert box with horizontal scrolling for Explanation Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove ExpandableNode and Add ListTree Updates the design for query explanation box, removes `ExpandableNode` and the dependency. Builts a new `ListTree` that does the same using reactstrap and custom css. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor refactor in Query API response `thanosInfo` is removed from Query reponse and used `explanation` directly. `disableCheckbox` is also renamed to `disableExplainCheckbox` in thanos UI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI tests to passing Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor UI changes and test fix UI improvements and Panel test fix other way around, resetting the results on panel construction. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update promql-engine to use Explain method Signed-off-by: Pradyumna Krishna <git@onpy.in> * Build UI assets Build UI assets, that runs new thanos UI with explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Revert proxy url change from package.json `proxy` was accidently changed and committed with package.json when removed dependency. Hence, reverting it back. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor changes in UI Fix requested changes in UI. - Rename `state` and `setState` to `mapping` and `setMapping`. - Rename `NodeTree` to `QueryTree`. - Use unicode characters instead of `-` and `+`. - Fix blue box on explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI assets Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (#6379) * Implementing Regex optimization on the MatchNotRegexp matcher type Signed-off-by: Alan Protasio <alanprot@gmail.com> * Opmizing MatchNotEqual Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Expected failure of block label query dedup tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Put back the correct makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * add de-cix as adopter (#6386) Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de> * Remove extra line that broke untouched test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add back line break at end of makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive single ingestor test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup issue in Receive Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add even more test cases for dedup on store gw Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup bug in Sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reuse nginx image name Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Let all users read the metrics file from static metrics server Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * [chore] Updating Query Engine and Prometheus (#6392) * Updating Query Engine Signed-off-by: Alan Protasio <alanprot@gmail.com> * fix prometheus breaking change Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update prometheus with https://github.com/prometheus/prometheus/pull/12387 Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Receive: Allow specifying tenant-specific external labels in RouterIngestor (#5777) Signed-off-by: haanhvu <haanh6594@gmail.com> * check context cancel when doing posting batches (#6396) Signed-off-by: Ben Ye <benye@amazon.com> * Expose store gateway query stats in series response hints (#6352) * expose query stats hints Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * add query stats hints in result Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * add merge method Signed-off-by: Ben Ye <benye@amazon.com> * fix unit test Signed-off-by: Ben Ye <benye@amazon.com> modify hints proto Signed-off-by: Ben Ye <benye@amazon.com> fix unit test Signed-off-by: Ben Ye <benye@amazon.com> update format Signed-off-by: Ben Ye <benye@amazon.com> * update comments Signed-off-by: Ben Ye <benye@amazon.com> * try again Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * receive: make az aware ketama hashring (#6369) * receive: make az aware ketama hashring Signed-off-by: Alexander Rickardsson <alxric@aiven.io> * receive: pass endpoints in hashring config as object Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive: add some tests for consistent hashing in presence of AZs Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive,docs: add migration note for az aware hashring Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> --------- Signed-off-by: Alexander Rickardsson <alxric@aiven.io> Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Reformat asciiflow chart Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Proposal: query path tenancy (#6320) * Add 1st version of query path tenancy proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update proposal after initial feedback Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add cool picture Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Include example in cross tenant query complications Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve reasoning for why not using the QFE Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve writing in "How" section Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix owner profile link Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address few more PR review comments Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address feedback on flag name text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve non-goals text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update docs/proposals-accepted/202304-query-path-tenancy.md Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify scenario for pitfalls of current solution Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify that Store doesn't care about tenant label Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add an action plan Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention alternative idea of modifying Store API Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address lots of feedback on the proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Format query path tenancy proposal doc Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add a "Tenancy Model" subsection to "Goals" Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention header semanthics in comparison with gRPC message field Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve action plan structure and writing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Fix double-counting bug in http_request_duration metric (#6399) * fix double-counting bug in http_request_duration metric Signed-off-by: 4orty <kwk5178@gmail.com> * Update Changelog Signed-off-by: 4orty <kwk5178@gmail.com> --------- Signed-off-by: 4orty <kwk5178@gmail.com> * Replace group with resolution in compact metrics. Compaction metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a `resolution` label to the compaction counters. Fixes: https://github.com/thanos-io/thanos/issues/5841 Signed-off-by: SuperQ <superq@gmail.com> * Reuse static metrics server from e2e framework Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Updates busybox SHA (#6403) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Fix series stats merge (#6408) * fix series stats merge Signed-off-by: Ben Ye <benye@amazon.com> * update license header Signed-off-by: Ben Ye <benye@amazon.com> * use reflect Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: allow unlimited head_series_limit tenants (#6406) With this commit we now allow to configure tenants with unlimited active series limit by setting the limit to `0`. Prior to this commit setting a per tenant limit to `0` would cause the tenant to be unable to write any metrics at all. This fixes: https://github.com/thanos-io/thanos/issues/6393 Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * expose downloaded data size in query hints (#6409) Signed-off-by: Ben Ye <benye@amazon.com> * maintainers: add myself to triagers (#6414) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Add `@douglascamata` to triagers (#6418) Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add Blog (#6411) * Add LFX blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add Headers to blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Lint blog Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * blog: Fix images for LFX post (#6422) * blog: Fix images for LFX post Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * fix lint Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Index Cache: Change cache key for postings (#6405) * extend postings cache key with codec Signed-off-by: Ben Ye <benye@amazon.com> * add changelog Signed-off-by: Ben Ye <benye@amazon.com> * update code back Signed-off-by: Ben Ye <benye@amazon.com> * add colon Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix another test Signed-off-by: Ben Ye <benye@amazon.com> * add compression scheme const to remote index cache Signed-off-by: Ben Ye <benye@amazon.com> * address required comments Signed-off-by: Ben Ye <benye@amazon.com> * fix compression scheme name Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: upgrading logs for failed uploads to error (#6427) * FIX: upgrading log for failed upload to error Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * docs: added changelog entry Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> --------- Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * fix postings test Signed-off-by: Ben Ye <benye@amazon.com> * Add aiven as adopter... more soon! Signed-off-by: Jonah Kowall <jkowall@kowall.net> * First draft of the Aiven adoption story Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add aiven as adopter... more soon! (#6430) Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Report gRPC connnection errors to the caller (#6428) By default `grpc.DialContext()` is non-blocking so any connection issue will not be surfaced to the user. This change makes it blocking and configures the gRPC dialer to report the underlying error if any happens. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * chore: remove duplicated `gopkg.in/fsnotify.v1` dep (#6432) * chore: remove duplicated `gopkg.in/fsnotify.v1` dep `github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same dependency. We can keep `github.com/fsnotify/fsnotify` and remove `gopkg.in/fsnotify.v1`. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * docs: add changelog Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> --------- Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * Expose estimated chunk and series size as configurable options (#6426) * expose estimated chunk and series size as configurable options Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: make tsdb stats limit configurable (#6437) * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> --------- Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * *: wire new Engine/Explain fields in query-frontend (#6433) - Pass Engine/Explain fields in query-frontend codecs - Add Engine field to QFE cache key - Add e2e tests for all cases Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index cache: Cache expanded postings (#6420) * cache expanded postings in index cache Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * rebase main and added compression name to key Signed-off-by: Ben Ye <benye@amazon.com> * update key Signed-off-by: Ben Ye <benye@amazon.com> * add e2e test for memcached Signed-off-by: Ben Ye <benye@amazon.com> * fix cache config Signed-off-by: Ben Ye <benye@amazon.com> * address review comments Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * add approximate series size to index stats (#6425) Signed-off-by: Ben Ye <benye@amazon.com> * index stats: fix chunk size calculation (#6424) Signed-off-by: Ben Ye <benye@amazon.com> * Remove some unused Cortex vendored code and metrics (#6440) * Avoid direct cortex vendor dependency in promclient (#6443) * use own explanation struct Signed-off-by: Ben Ye <benye@amazon.com> * omit empty Signed-off-by: Ben Ye <benye@amazon.com> * fix e2e test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * query.md: Added grpc_{client,server}_handled_total to metrics list. (#6351) Fixes: #6280 Signed-off-by: Jonathan Davies <jpds@protonmail.com> * Updates busybox SHA (#6448) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * store: read postings directly into delta encoded format (#6442) Instead of allocating bytes for raw postings, let's read them directly into diff varint format to save memory. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * compact: Add Index Stats to block metadata (#6441) * compactor set index stats on block Signed-off-by: Ben Ye <benye@amazon.com> * comment Signed-off-by: Ben Ye <benye@amazon.com> * change field Signed-off-by: Ben Ye <benye@amazon.com> * update downsample Signed-off-by: Ben Ye <benye@amazon.com> * add index stats into compaction Signed-off-by: Ben Ye <benye@amazon.com> * fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * estimate block chunk and series size from metadata (#6449) Signed-off-by: Ben Ye <benye@amazon.com> * Adding graphic for blog showing architecture Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing mermaid graph Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing mermaid graph again Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Remove image and fix mermaid for other flowchart Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing prod flowchart Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing prod flowchart title Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing md table Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing md table again Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Store: fix crash on empty regex matcher Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Some corrections on text and links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing diagams Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing currencies Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing diagams Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing images Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Website: Fix blog images width (#6450) * Set blog post images width to 100% Blog Post images width isn't set and hence the images doesn't get fit to the page. Setting width to 100% make sure images fit to the web page and screen size. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove width constraint from blog images Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Cache calculated mint and maxt for each remote engine (#6458) The distributed optimizer in the Thanos engine calls MinT() and LabelSets multiple times on each remote engine. These calculations can get expensive when an engine covers a large amount of TSDBs. This commit introduces caching of the calculated values for a remote engine's mint, maxt and label sets. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cache empty expanded postings (#6464) Signed-off-by: Ben Ye <benye@amazon.com> * mixin/receive: add limits alerting (#6466) * add some thanos receive limits alerts Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix and generate examples Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * quick fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * quick fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix test Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix invalid changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update docs/blog/2023-06-08-thanos-at-aiven.md Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add retention for M3 Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add image versus mermaid diagram. Couldn't comment it out unfortunately Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding text around link to fix linter Signed-off-by: Jonah Kowall <jkowall@kowall.net> * make docs Signed-off-by: Jonah Kowall <jkowall@kowall.net> * add alert for tenant reaching head series limit (#6467) Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * go.mod: bump rest of otel libs (#6447) * go.mod: bump rest of otel libs Bump these two OTEL libs to be like the rest of the OTEL libs. We had a strange issue where `service.name` was not set even though it was in environment variables. Bumping these libs fixes the problem. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * otlp: only set service.name if provided Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * e2e: add test case Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: disable pooling for postings benchmarks (#6473) Pooling hides the real allocations in postings benchmarks because the allocation only happens once really. In reality this is different when multiple readers are created at once. Disable pooling in benchmarks to understand what is happening. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Fixing date and author for Aiven blog (#6476) * Fixing authors Signed-off-by: Jonah Kowall <jkowall@kowall.net> * make docs and fixing date Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing authors Signed-off-by: Jonah Kowall <jkowall@kowall.net> --------- Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Check context when expanding postings 2nd attempt (#6471) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * import Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * [FEAT] Add thanos ruler blog post (#6477) Signed-off-by: Hélia Barroso <helia.barroso@farfetch.com> * blog: Small fixes to Aiven article (#6481) * Correct some links and typos in 2023-06-08-thanos-at-aiven.md The linkified URLs in the article were including the closing bracket, breaking the links. Make them proper Markdown links. Also fixed a few other typos I spotted. Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> * Update 2023-06-08-thanos-at-aiven.md Clarify a couple of things in the final paragraph. Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> --------- Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> * Rename series and postings fetch duration metrics (#6479) * fix wrong metric names for series and postings fetch duration Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * note breaking change Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Update to prometheus/common v0.44.0 (#6483) Signed-off-by: Dan Bason <colonel.dan@gmail.com> * go.mod: update rueidis client (#6485) The latest improvements have great perf improvements. Let's update it. Thank you to [rueian](https://github.com/rueian) and all of the contributors to that awesome Redis client! Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Introduce tenancy package (#6482) This commit introduces a tenancy package in preparation for introduction of tenancy in the query path. Tenancy related code which can be common between components are moved out of the receive component into the tenancy package. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Ignore nginx.com links in mdox (#6489) Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * blog: Update Aiven blog post (#6487) Wording changes and better formatting for the tables. Signed-off-by: Simon Brüggen <simon.bruggen@aiven.io> * store: optimized snappy streamed reading (#6475) Optimize snappy streamed reading by traversing through the byte slice in advance to determine how big of a buffer we will need. I have opted to rewrite snappy streamed format decoding because it is a straightforward format. Complex parts were deferred to klauspost/compress. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Remove Exists call in meta fetcher (#6474) * Remove Exists call in meta fetcher The meta fetcher lists the bucket by iterating through top level paths from the entire keyspace, and then for each key calls Exists on the meta.json file. This leads to an N+1 amplification against object storage, manifesting as a very high increase in costs and occasional throttling by cloud providers. This commit changes the iteration logic to recursively list all files instead, which can be done with far fewer API calls to object storage. Meta files are then identified from the keys returned by the list operation instead of making one API request from each file. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Revert e2e test change Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * e2e/store: use now instead of time.Now() each time (#6493) Consistently use the same timestamp instead of the current time each time during a test. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * [FIX] Documentation issue (#6494) Signed-off-by: Hélia Barroso <helia_barroso@hotmail.com> * Updates busybox SHA (#6497) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * chore: pkg imported more than once (#6499) * chore: pkg imported more than once Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> * fix: remove pkg alias Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> --------- Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> * *: Remove unnecessary configuration reload from `ContentPathReloader` and improve its tests (#6496) * Fix and improve PathContentReloader tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run tests in parallel Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add changelog entry Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix table tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> * add context check when decoding cached postings (#6506) Signed-off-by: Ben Ye <benye@amazon.com> * *: Change the engine behind `ContentPathReloader` to be completely independent of any filesystem concept. (#6503) * Add fallback config reload for symlinks Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve pooling config reload Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make tests more reliable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve tests again Assert on runutil.Repeat output Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Pass the debounce/reload time to each test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add comments and ensure interfaces are implemented Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make the limit configuration reload time customizable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * goimports file Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix lint warning Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve log for polling engine Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make linter happy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Extract symlink identification to a separate function Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix links to go-grpc-middleware after v2 merge into main Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add period to make linter happy. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog entry Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Replace the fsnotify based engine from `PathContentReloader` with the polling one Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove check for empty filePath Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Unexpose pollingEngine.Start Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix pollingEngine doc-comment Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Log file path when config is reloaded Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * receive: add float histogram support (#6323) * Added receive float histogram support Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added comments for fns copied from Prometheus Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Improved comment Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unnecessary if Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed native histogram proto conversion in remote engine Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused histogram conversion from remote engine Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fix and renaming in native_histograms_test.go Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Trigger Build Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * go.mod: bump e2e framework version (#6516) Bump e2e framework version and add a note about E2E_DOCKER_CPUS. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Add changelog for #6474 (#6518) Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cortex/redisclient: use rueidis client (#6520) * cortex/redisclient: use rueidis client Use the same rueidis client in query-frontend. Solves https://github.com/thanos-io/thanos/issues/6094. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cortex/cache: gofumpt + fix errors.Errorf Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil/docs: clean up more old stuff Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index header: Remove memWriter from fileWriter (#6509) * index header: remove memWriter from fileWriter Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * refactor Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * update comment Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Wrap object store Prometheus registry (#6152) * Wrap object store Prometheus registery In preparation for the work being done at https://github.com/thanos-io/objstore/pull/26. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update objstore to the latest version This version removes the `thanos_` prefix from metrics, which is the reason for wrapping the objstore's metrics registry in the first place. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix modules Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Wrap metrics registerer for objstore bucket Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove prefix from Thanos Store metrics Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix goimports Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Put back upgraded objstore dep Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Move to more recent ref of thanos/objstore Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Ignore OBS in objstore tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix linting error on main test after objstore upgrade Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Skip OCS objstore test in circle ci Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix echo in makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * e2e/store: try to fix Se…

@damnever

…hanos-io#199) * Replace summary in extprom metrics with histogram (thanos-io#6327) * Replaced summary in extprom metrics with histogram Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused parameters from NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Reverted NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Avoid expensive log.Valuer evaluation for disallowed levels (thanos-io#6322) Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Fix inconsistent error for series limits in Store API (thanos-io#6330) * store: fix inconsistent error for series limits Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * rename labelValues serires liimiter test function Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * *: remove unmaintained gzip library (thanos-io#6332) Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Traces sampler env var (thanos-io#6306) * Issue#5947 OTEL_TRACES_SAMPLER env var Signed-off-by: shayyxi <shazi12384@gmail.com> * Test correction Signed-off-by: shayyxi <shazi12384@gmail.com> * doc failure correction. parse float argument correction. Signed-off-by: shayyxi <shazi12384@gmail.com> * added the changelog. Signed-off-by: shayyxi <shazi12384@gmail.com> * ran make docs to fix the build failure. Signed-off-by: shayyxi <shazi12384@gmail.com> * corrected the incorrect change in tools.md Signed-off-by: shayyxi <shazi12384@gmail.com> * fixed review comments. Signed-off-by: shayyxi <shazi12384@gmail.com> --------- Signed-off-by: shayyxi <shazi12384@gmail.com> Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com> Co-authored-by: shayyxi <shazi12384@gmail.com> * query: use storepb.SeriesServer (thanos-io#6334) Use storepb.SeriesServer instead of the concrete struct. This allows implementing functionality on top of the proxy. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. redis/rueidis#209 (thanos-io#6342) * use github.com/onsi/gomega/gleak to detect goroutine leak with timeout Signed-off-by: Rueian <rueiancsie@gmail.com> * fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (thanos-io#6345) Signed-off-by: aimuz <mr.imuz@gmail.com> * Return grpc code resource exhausted for byte limit error (thanos-io#6325) * return grpc code resource exhausted for byte limit error Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * update partial response strategy Signed-off-by: Ben Ye <benye@amazon.com> * fix limit Signed-off-by: Ben Ye <benye@amazon.com> * try to fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test error message Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Expose info for each TSDB This commit exposes the label set alongside the min and max time for each TSDB covered by a Store. This information is used to scope the min time for a remote query so that we do not produce partial aggregates in distriuted mode. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add test case for proxy store Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Bump promql-engine to fix thanos-io/promql-engine#239 (thanos-io#6349) Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> * Updates busybox SHA (thanos-io#6365) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Query: Add +Inf bucket to query duration metrics (thanos-io#6358) * Query: Add +Inf bucket to query duration metrics For the query duration metrics (`thanos_store_api_query_duration_seconds`), we record query respond latency, based on the size of the query (samples/series), and save to a histogram. However, when a query is made which exceeds the biggest sample/serie size, we would prior to this commit, put the request into the largest bucket. With this commit, we instead create an `+Inf` bucket, and put requests which are larger than the biggest defined bucket into that. This gives more accurate results, and also allow one to see if the bucket sizes are incorrectly sized. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Tests: Mutex around non-thread safe random source When creating test blocks, we use a non-thread safe random source, in multiple goroutines. Due to this, tests would sometime panic. This commits puts a mutex around calls using the same source, in order to avoid this. This should hopefully improve reliability of e2e tests. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * e2e(query): Reproduce dedup issue from thanos-io#6257 Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add dedup e2e test for Receive With internal and external labels support. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Simplify generated blocks for query test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve query dedup test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Write a query test for dedup with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Refactor query dedup test with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive query test Now it properly ensures the double dedup works (on internal and external labels). Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix receive drawing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add one extra test caes for query dedup from store Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Receive query with dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Sidecar query dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Expected failure of block label query dedup tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Check context when expanding postings (thanos-io#6363) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * ui: only keep name in store_matches param (thanos-io#6371) We are doing store matching on the `name` field hence only keep that field in the URL because otherwise the URL could get quite lengthy with external labelsets inside of it. Besides unit tests, I have also tested locally: - Enable store filtering; - Select store(-s); - Copy/paste URL into the new tab and see that the same stores are loaded like expected; - See that URL only has names in them. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * docs: replace --store with --endpoint Replace deprecated `--store` with `--endpoint` in docs. Signed-off-by: Paul Gier <paul.gier@datastax.com> * Optimizing "grafana generated" regex matchers (thanos-io#6376) * Opmizing Group Regex Signed-off-by: Alan Protasio <alanprot@gmail.com> * fixing native histogram tests Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Cache: various index cache client improvements (thanos-io#6374) * Query Explanation (thanos-io#6346) * Return Query Explaination in QueryAPI A param `explain` is added to QueryAPI, if true then explanation returned by the `Explain()` method of the query having structure `ExplainOutputNode` is returned in response. Query Explanation is added under new field in response that is `thanosInfo`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add explain checkbox in thanos UI A explain checkbox is added to Thanos Query UI, that requests for query explanation from thanos query api. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add ExpandableNode Component ExpandableNode component renders Query Explanation in the thanos UI. Requires a new package `react-accessible-treeview`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Disable Explain checkbox on prometheus engine Prometheus engine sends out error if toggle explain button. To provide better experience, the explain checkbox get disbaled on switching to prometheus engine and enable back on switching to thanos engine. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add alert box with horizontal scrolling for Explanation Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove ExpandableNode and Add ListTree Updates the design for query explanation box, removes `ExpandableNode` and the dependency. Builts a new `ListTree` that does the same using reactstrap and custom css. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor refactor in Query API response `thanosInfo` is removed from Query reponse and used `explanation` directly. `disableCheckbox` is also renamed to `disableExplainCheckbox` in thanos UI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI tests to passing Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor UI changes and test fix UI improvements and Panel test fix other way around, resetting the results on panel construction. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update promql-engine to use Explain method Signed-off-by: Pradyumna Krishna <git@onpy.in> * Build UI assets Build UI assets, that runs new thanos UI with explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Revert proxy url change from package.json `proxy` was accidently changed and committed with package.json when removed dependency. Hence, reverting it back. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor changes in UI Fix requested changes in UI. - Rename `state` and `setState` to `mapping` and `setMapping`. - Rename `NodeTree` to `QueryTree`. - Use unicode characters instead of `-` and `+`. - Fix blue box on explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI assets Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (thanos-io#6379) * Implementing Regex optimization on the MatchNotRegexp matcher type Signed-off-by: Alan Protasio <alanprot@gmail.com> * Opmizing MatchNotEqual Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Put back the correct makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove extra line that broke untouched test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add back line break at end of makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive single ingestor test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup issue in Receive Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add even more test cases for dedup on store gw Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup bug in Sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reuse nginx image name Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Let all users read the metrics file from static metrics server Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reformat asciiflow chart Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reuse static metrics server from e2e framework Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * add de-cix as adopter (thanos-io#6386) Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de> * [chore] Updating Query Engine and Prometheus (thanos-io#6392) * Updating Query Engine Signed-off-by: Alan Protasio <alanprot@gmail.com> * fix prometheus breaking change Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update prometheus with prometheus/prometheus#12387 Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Receive: Allow specifying tenant-specific external labels in RouterIngestor (thanos-io#5777) Signed-off-by: haanhvu <haanh6594@gmail.com> * check context cancel when doing posting batches (thanos-io#6396) Signed-off-by: Ben Ye <benye@amazon.com> * Expose store gateway query stats in series response hints (thanos-io#6352) * expose query stats hints Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * add query stats hints in result Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * add merge method Signed-off-by: Ben Ye <benye@amazon.com> * fix unit test Signed-off-by: Ben Ye <benye@amazon.com> modify hints proto Signed-off-by: Ben Ye <benye@amazon.com> fix unit test Signed-off-by: Ben Ye <benye@amazon.com> update format Signed-off-by: Ben Ye <benye@amazon.com> * update comments Signed-off-by: Ben Ye <benye@amazon.com> * try again Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * receive: make az aware ketama hashring (thanos-io#6369) * receive: make az aware ketama hashring Signed-off-by: Alexander Rickardsson <alxric@aiven.io> * receive: pass endpoints in hashring config as object Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive: add some tests for consistent hashing in presence of AZs Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive,docs: add migration note for az aware hashring Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> --------- Signed-off-by: Alexander Rickardsson <alxric@aiven.io> Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Proposal: query path tenancy (thanos-io#6320) * Add 1st version of query path tenancy proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update proposal after initial feedback Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add cool picture Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Include example in cross tenant query complications Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve reasoning for why not using the QFE Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve writing in "How" section Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix owner profile link Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address few more PR review comments Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address feedback on flag name text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve non-goals text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update docs/proposals-accepted/202304-query-path-tenancy.md Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify scenario for pitfalls of current solution Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify that Store doesn't care about tenant label Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add an action plan Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention alternative idea of modifying Store API Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address lots of feedback on the proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Format query path tenancy proposal doc Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add a "Tenancy Model" subsection to "Goals" Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention header semanthics in comparison with gRPC message field Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve action plan structure and writing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Fix double-counting bug in http_request_duration metric (thanos-io#6399) * fix double-counting bug in http_request_duration metric Signed-off-by: 4orty <kwk5178@gmail.com> * Update Changelog Signed-off-by: 4orty <kwk5178@gmail.com> --------- Signed-off-by: 4orty <kwk5178@gmail.com> * Updates busybox SHA (thanos-io#6403) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Fix series stats merge (thanos-io#6408) * fix series stats merge Signed-off-by: Ben Ye <benye@amazon.com> * update license header Signed-off-by: Ben Ye <benye@amazon.com> * use reflect Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: allow unlimited head_series_limit tenants (thanos-io#6406) With this commit we now allow to configure tenants with unlimited active series limit by setting the limit to `0`. Prior to this commit setting a per tenant limit to `0` would cause the tenant to be unable to write any metrics at all. This fixes: thanos-io#6393 Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * expose downloaded data size in query hints (thanos-io#6409) Signed-off-by: Ben Ye <benye@amazon.com> * maintainers: add myself to triagers (thanos-io#6414) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Add `@douglascamata` to triagers (thanos-io#6418) Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add Blog (thanos-io#6411) * Add LFX blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add Headers to blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Lint blog Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * blog: Fix images for LFX post (thanos-io#6422) * blog: Fix images for LFX post Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * fix lint Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Index Cache: Change cache key for postings (thanos-io#6405) * extend postings cache key with codec Signed-off-by: Ben Ye <benye@amazon.com> * add changelog Signed-off-by: Ben Ye <benye@amazon.com> * update code back Signed-off-by: Ben Ye <benye@amazon.com> * add colon Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix another test Signed-off-by: Ben Ye <benye@amazon.com> * add compression scheme const to remote index cache Signed-off-by: Ben Ye <benye@amazon.com> * address required comments Signed-off-by: Ben Ye <benye@amazon.com> * fix compression scheme name Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: upgrading logs for failed uploads to error (thanos-io#6427) * FIX: upgrading log for failed upload to error Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * docs: added changelog entry Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> --------- Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * fix postings test Signed-off-by: Ben Ye <benye@amazon.com> * Add aiven as adopter... more soon! (thanos-io#6430) Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Report gRPC connnection errors to the caller (thanos-io#6428) By default `grpc.DialContext()` is non-blocking so any connection issue will not be surfaced to the user. This change makes it blocking and configures the gRPC dialer to report the underlying error if any happens. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * chore: remove duplicated `gopkg.in/fsnotify.v1` dep (thanos-io#6432) * chore: remove duplicated `gopkg.in/fsnotify.v1` dep `github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same dependency. We can keep `github.com/fsnotify/fsnotify` and remove `gopkg.in/fsnotify.v1`. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * docs: add changelog Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> --------- Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * Expose estimated chunk and series size as configurable options (thanos-io#6426) * expose estimated chunk and series size as configurable options Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: make tsdb stats limit configurable (thanos-io#6437) * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> --------- Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * *: wire new Engine/Explain fields in query-frontend (thanos-io#6433) - Pass Engine/Explain fields in query-frontend codecs - Add Engine field to QFE cache key - Add e2e tests for all cases Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index cache: Cache expanded postings (thanos-io#6420) * cache expanded postings in index cache Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * rebase main and added compression name to key Signed-off-by: Ben Ye <benye@amazon.com> * update key Signed-off-by: Ben Ye <benye@amazon.com> * add e2e test for memcached Signed-off-by: Ben Ye <benye@amazon.com> * fix cache config Signed-off-by: Ben Ye <benye@amazon.com> * address review comments Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * add approximate series size to index stats (thanos-io#6425) Signed-off-by: Ben Ye <benye@amazon.com> * index stats: fix chunk size calculation (thanos-io#6424) Signed-off-by: Ben Ye <benye@amazon.com> * Remove some unused Cortex vendored code and metrics (thanos-io#6440) * Fixed DefaultPromConfig * Fixed imports * Back to diffVarintSnappyEncode * Merge pull request thanos-io#180 from Shopify/optimize-timerange-calculation Cache calculated mint and maxt for each remote engine * Updated busybox * fixing lint * Fixing merge conflict Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Fixing missing import Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fix lint again Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * resolving conflict merges Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Fixed import and fn order * Fixed unit tests * Updated promdoc.sum * Back to custom promql engine * Removed custom promql engine and moved to latest upstream * Ran go mod tidy * Fixed GetQueryAPIClients * Store: fix crash on empty regex matcher Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: shayyxi <shazi12384@gmail.com> Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com> Signed-off-by: Rueian <rueiancsie@gmail.com> Signed-off-by: aimuz <mr.imuz@gmail.com> Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Signed-off-by: Paul Gier <paul.gier@datastax.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> Signed-off-by: Pradyumna Krishna <git@onpy.in> Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de> Signed-off-by: haanhvu <haanh6594@gmail.com> Signed-off-by: Alexander Rickardsson <alxric@aiven.io> Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> Signed-off-by: 4orty <kwk5178@gmail.com> Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> Signed-off-by: Jonah Kowall <jkowall@kowall.net> Signed-off-by: Simon Pasquier <spasquie@redhat.com> Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Co-authored-by: Sebastian Rabenhorst <4246554+rabenhorst@users.noreply.github.com> Co-authored-by: Xiaochao Dong <the.xcdong@gmail.com> Co-authored-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Co-authored-by: Shazi <42436533+shayyxi@users.noreply.github.com> Co-authored-by: shayyxi <shazi12384@gmail.com> Co-authored-by: Rueian <rueiancsie@gmail.com> Co-authored-by: aimuz <mr.imuz@gmail.com> Co-authored-by: Ben Ye <benye@amazon.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Alban Hurtaud <alban.hurtaud@amadeus.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> Co-authored-by: Jacob Baungård Hansen <jacobbaungard@redhat.com> Co-authored-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Paul Gier <paul.gier@datastax.com> Co-authored-by: Alan Protasio <alanprot@gmail.com> Co-authored-by: Pradyumna Krishna <git@onpy.in> Co-authored-by: Raúl Garcia Sanchez <info@raulgarcia.de> Co-authored-by: Ha Anh Vu <75315486+haanhvu@users.noreply.github.com> Co-authored-by: Alexander Rickardsson <alxric@aiven.io> Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io> Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Co-authored-by: Wonki Kim <kwk5178@gmail.com> Co-authored-by: Michael Hoffmann <mhoffm@posteo.de> Co-authored-by: Victor Hugo Brito Fernandes <victorhbfernandes@gmail.com> Co-authored-by: Jonah Kowall <jkowall@kowall.net> Co-authored-by: Simon Pasquier <spasquie@redhat.com> Co-authored-by: Eng Zer Jun <engzerjun@gmail.com> Co-authored-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Add 1st version of query path tenancy proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update proposal after initial feedback Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add cool picture Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Include example in cross tenant query complications Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve reasoning for why not using the QFE Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve writing in "How" section Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix owner profile link Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address few more PR review comments Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address feedback on flag name text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve non-goals text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update docs/proposals-accepted/202304-query-path-tenancy.md Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify scenario for pitfalls of current solution Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify that Store doesn't care about tenant label Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add an action plan Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention alternative idea of modifying Store API Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address lots of feedback on the proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Format query path tenancy proposal doc Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add a "Tenancy Model" subsection to "Goals" Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention header semanthics in comparison with gRPC message field Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve action plan structure and writing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com>

douglascamata added 6 commits April 26, 2023 10:32

Add 1st version of query path tenancy proposal

2b86f4b

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Update proposal after initial feedback

8a9ffa3

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Add cool picture

38c85c1

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Include example in cross tenant query complications

2873c12

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Improve reasoning for why not using the QFE

d68586b

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Improve writing in "How" section

908accb

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

pull-request-size bot added the size/L label Apr 26, 2023

Fix owner profile link

52c865e

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

saswatamcode reviewed Apr 27, 2023

View reviewed changes

douglascamata and others added 6 commits April 27, 2023 10:45

Apply suggestions from code review

dc9b64b

Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Address few more PR review comments

aa8f20a

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Address feedback on flag name text

ec47877

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Update diagram

b14982c

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Improve non-goals text

1ba8493

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Update diagram

abc44d8

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

pull-request-size bot added size/M and removed size/L labels Apr 27, 2023

fpetkovski reviewed Apr 27, 2023

View reviewed changes

douglascamata and others added 5 commits April 27, 2023 14:40

Update docs/proposals-accepted/202304-query-path-tenancy.md

d325493

Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Clarify scenario for pitfalls of current solution

f7d1dc0

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Clarify that Store doesn't care about tenant label

5e12994

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Add an action plan

c95450e

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Mention alternative idea of modifying Store API

c2e5f0e

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

pull-request-size bot added size/L and removed size/M labels Apr 28, 2023

Merge branch 'main' of github.com:thanos-io/thanos into query-path-te…

e9a6fa2

…nancy-proposal

fpetkovski reviewed May 2, 2023

View reviewed changes

bwplotka reviewed May 3, 2023

View reviewed changes

GiedriusS reviewed May 4, 2023

View reviewed changes

docs/proposals-accepted/202304-query-path-tenancy.md Outdated Show resolved Hide resolved

GiedriusS reviewed May 4, 2023

View reviewed changes

douglascamata and others added 6 commits May 4, 2023 11:25

Fix typo

cf71fb6

Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Address lots of feedback on the proposal

0001b62

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Format query path tenancy proposal doc

cd34490

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Add a "Tenancy Model" subsection to "Goals"

98b902b

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Mention header semanthics in comparison with gRPC message field

f80384e

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Improve action plan structure and writing

300ed71

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

saswatamcode approved these changes May 30, 2023

View reviewed changes

fpetkovski approved these changes May 30, 2023

View reviewed changes

saswatamcode merged commit 03aea4f into thanos-io:main May 30, 2023

jacobbaungard mentioned this pull request Jul 18, 2023

Query: Forward tenant information via StoreAPI #6530

Merged

2 tasks


		## How

		* Implement a mechanism to allow incoming requests to specify the tenant being queried using an HTTP header. This applies to all components in the query path. We follow the example of Thanos Receive, which uses the `--receive.tenant-header="THANOS-TENANT"` flag to configure the tenant header, adapting its name to each component. So in the Querier, the flag name will be `--querier.tenant-header`, in the Store Gateway it'll be `--store.tenant-header`, and so on.

Conversation

douglascamata commented Apr 26, 2023

Uh oh!

saswatamcode left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bwplotka May 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

douglascamata May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

douglascamata May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fpetkovski May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bwplotka May 3, 2023 •

edited

Loading

douglascamata May 2, 2023 •

edited

Loading

douglascamata May 2, 2023 •

edited

Loading

fpetkovski May 2, 2023 •

edited

Loading

bwplotka May 3, 2023 •

edited

Loading

douglascamata May 4, 2023 •

edited

Loading