Skip to content

Engine issue: different results with thanos and prometheus engines. group_left? #239

@ahurtaud

Description

@ahurtaud

Hello,

We have 2 similar thanos query (observee / observer) architectures. (query->queriers->sidecars):
One with the prometheus promql engine and one with the thanos promql engine and the following query gave different results with the thanos promql engines compared to thanos with prometheus engine and directly executed in prometheus also:

avg by (hrStorageDescr) (hrStorageUsed{nodename =~"xxxxx", } * on (instance, hrStorageIndex ) group_left(hrStorageDescr) (sum by (instance, hrStorageIndex, hrStorageDescr) (hrStorageDescr{nodename =~"xxxxx", hrStorageDescr!~".*Data.*"} ) ))

results with thanos engine (wrong one):
Screenshot 2023-04-12 at 16 44 51

results with prometheus engine (ok one):
Screenshot 2023-04-12 at 16 47 38

results in prometheus ui directly:
Screenshot 2023-04-12 at 16 48 32

Sorry I had no time yet to try to reproduce with a "generic" exporter/query. I may try in the coming days.
I suspect group_left queries (maybe only with negative matchers?). Are those supported by the engine yet?

Thanks

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions