Skip to content

Add query watches api to retrieve multiple watches.#64582

Merged
martijnvg merged 18 commits intoelastic:masterfrom
martijnvg:watcher_list_api
Dec 3, 2020
Merged

Add query watches api to retrieve multiple watches.#64582
martijnvg merged 18 commits intoelastic:masterfrom
martijnvg:watcher_list_api

Conversation

@martijnvg
Copy link
Copy Markdown
Member

@martijnvg martijnvg commented Nov 4, 2020

This api supports pagination (from / size) and
querying and sorting by watch _id and watcher metadata.

This avoids using .watch index directly.
On a per watch basis the same information that the get watch api returns is returned,
except version.

Example 1:

GET /_watcher/_query_watches
{
    "count": 2,
    "watches": [
        {
            "_id": "my-watch1",
            "watch": {
                "trigger": {
                    "schedule": {
                        "yearly": [
                            {
                                "in": [
                                    "JAN"
                                ],
                                "on": [
                                    10
                                ],
                                "at": [
                                    "noon"
                                ]
                            }
                        ]
                    }
                },
                "input": {
                    "simple": {}
                },
                "condition": {
                    "never": {}
                },
                "actions": {}
            },
            "status": {
                "state": {
                    "active": true,
                    "timestamp": "2020-11-04T11:49:50.891Z"
                },
                "actions": {},
                "version": -1
            },
            "_seq_no": 0,
            "_primary_term": 1
        },
        {
            "_id": "my-watch2",
            "watch": {
                "trigger": {
                    "schedule": {
                        "yearly": [
                            {
                                "in": [
                                    "JAN"
                                ],
                                "on": [
                                    10
                                ],
                                "at": [
                                    "noon"
                                ]
                            }
                        ]
                    }
                },
                "input": {
                    "simple": {}
                },
                "condition": {
                    "never": {}
                },
                "actions": {}
            },
            "status": {
                "state": {
                    "active": true,
                    "timestamp": "2020-11-04T11:49:53.500Z"
                },
                "actions": {},
                "version": -1
            },
            "_seq_no": 1,
            "_primary_term": 1
        }
    ]
}

Example 2:

GET /_watcher/_list_watches
{
    "query": {
        "term": {
            "_id": "my-watch2"
        }
    }
}
{
    "count": 1,
    "watches": [
        {
            "_id": "my-watch2",
            "watch": {
                "trigger": {
                    "schedule": {
                        "yearly": [
                            {
                                "in": [
                                    "JAN"
                                ],
                                "on": [
                                    10
                                ],
                                "at": [
                                    "noon"
                                ]
                            }
                        ]
                    }
                },
                "input": {
                    "simple": {}
                },
                "condition": {
                    "never": {}
                },
                "actions": {}
            },
            "status": {
                "state": {
                    "active": true,
                    "timestamp": "2020-11-04T11:49:53.500Z"
                },
                "actions": {},
                "version": -1
            },
            "_seq_no": 1,
            "_primary_term": 1
        }
    ]
}

Relates #62501

This api supports pagination (from / size) and
querying and sorting by watch _id and watcher metadata.

This avoids using .watch index directly.

Relates elastic#62501
@elasticmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Watcher)

@elasticmachine elasticmachine added the Team:Data Management (obsolete) DO NOT USE. This team no longer exists. label Nov 4, 2020
@cjcenizal
Copy link
Copy Markdown
Contributor

cjcenizal commented Nov 10, 2020

@martijnvg I think this API will work for the Watcher UI, but I wanted to do a bit of a brain dump, both to keep a record of my thoughts and also to get your eyes on my thought process in case I'm missing or misunderstanding anything.

Scrolling and pagination

Currently, Watcher fetches watches by specifying scroll: "30s" and body.size: 100 parameters in the request. It then passes the response to a fetchAllFromScroll utility that recursively checks the results for hits.hits and calls the Scroll API if this value is greater than 0. It returns the accumulated hits from all scroll search results. I'm guessing the intention behind this logic is to retrieve a large number of watches without timing out the request (which is a potential misuse of the Scroll API but that's not germane to this discussion).

My question is: can we replace this logic with something similar using the from and size pagination parameters? I believe the answer is yes. We'd have a fetchAllWatches utility that looks something like this (code untested):

export const fetchAllWatches = async (
  dataClient: ILegacyScopedClusterClient,
  from: number,
  size: number,
  accumulatedHits: any[] = []
): Promise<any> => {
  const response = await dataClient.callAsCurrentUser('watcher.listWatches', {
    body: {
      from,
      size,
    },
    ignore: [404],
  });

  const hits = get(response, 'hits.hits', []);
  accumulatedHits.push(...hits);

  // If we asked for a number of results equal to `size`, and got back that amount, then there
  // may be more to retrieve.
  if (hits.length === size) {
    return fetchAllWatches(dataClient, from + size, size, accumulatedHits);
  }

  return accumulatedHits;
};

// Begin series of paginated requests.
fetchAllWatches(dataClient, 0, 100, []);

I'm basing this off the search pagination docs. This is generally how pagination works with the from and size parameters, right?

Ignoring 404s

The original fetchAllFromScroll utility and the above fetchAllWatches utility configure requests with ignore: [404]. This is documented in the JS client docs as a means of allowing the request to proceed and resolve to a 200 even if it encounters a 404 as part of its operation. @martijnvg Is this part of the ES API or just part of the JS client?

@martijnvg
Copy link
Copy Markdown
Member Author

@cjcenizal This api is a wrapper around the _search api. Supporting scroll would mean we would also need to write a wrapper around the search scroll api. This felt a bit too much to me, sense a scroll search tends to be used when paging through large amounts of documents and this doesn't apply to watches. I think we can expose search after in the list watcher api, which allows to page through the watches in a convenient and efficient way like scroll, but this can then all be done via the list watches api. Would that make things easier on the UI side?

Is this part of the ES API or just part of the JS client?

I don't that particular option is part of ES. In ES there is ignore_unavailable option to avoid failure in case specified indices don't exist. I think we can expose that option in this API as well.

@cjcenizal
Copy link
Copy Markdown
Contributor

Thanks for the context @martijnvg. When we talk about scroll being suited for "large numbers of documents", what's the rough order of magnitude we have in mind? Tens of thousands? Millions?

The PR description mentions that this API supports from and size for pagination, so the code example I gave depends on that. Do you see a problem with using from and size? Did you suggest search_after because it's better suited for this type of use case (bulk retrieval rather than true paginated consumption)? I'm not familiar with either option, so I need guidance.

In terms of ignore_unavailable, I don't see any reason not to support it though I don't think we will consume it. I don't see how a 404 can occur when fetching all watches. In the situation we fetch an individual watch, a 404 error is helpful for the user.

Also, I just wanted to note that we're using scroll to fetch the .watcher-history-* documents as well.

@martijnvg
Copy link
Copy Markdown
Member Author

Thanks for the context @martijnvg. When we talk about scroll being suited for "large numbers of documents", what's the rough order of magnitude we have in mind? Tens of thousands? Millions?

I think in most cases there will not be more than a few thousands watches. So I think regular pagination should be fine.

Did you suggest search_after because it's better suited for this type of use case (bulk retrieval rather than true paginated consumption)?

Search after is like scroll search in the sense that it is optimised for deep pagination. With the sort values of the last hit, you can paginate through all the hits like is possible with from / size pagination. The main difference with scroll and search after is that scroll search takes into account the latest updates to an index and therefor the sort order may change during pagination, whereas with scroll search the view of the index remain the same during the lifetime of a scroll id. However search after can be used in combination with the point in time api to preserve the state of an index while searches happen to paginate though the results.

In terms of ignore_unavailable, I don't see any reason not to support it though I don't think we will consume it. I don't see how a 404 can occur when fetching all watches. In the situation we fetch an individual watch, a 404 error is helpful for the user.

I see. I think just returning an empty response is ok when there are no watches or there is no .watches index.

@cjcenizal
Copy link
Copy Markdown
Contributor

Thanks Martijn! This is great info. search_after in conjunction with pit sounds perfectly suited for this particular use case, which is more of a bulk retrieval than standard pagination.

scroll search takes into account the latest updates to an index and therefor the sort order may change during pagination, whereas with scroll search the view of the index remain the same during the lifetime of a scroll id

Did you mean to say "search after takes into account the latest updates to an index"?

I see. I think just returning an empty response is ok when there are no watches or there is no .watches index.

Agreed! This is much more ergonomic for us as consumers. The absence of watches or a .watches index is an implementation detail from the UI's perspective -- all it cares about is the watches that are available, with 0 being a perfectly valid answer.

@martijnvg
Copy link
Copy Markdown
Member Author

Did you mean to say "search after takes into account the latest updates to an index"?

Whoops, yes, this is what I meant.

@martijnvg martijnvg requested a review from jakelandis November 19, 2020 15:58
@martijnvg
Copy link
Copy Markdown
Member Author

@elasticmachine run elasticsearch-ci/docs

Copy link
Copy Markdown
Contributor

@jakelandis jakelandis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _id field is only a keyword, and the metadata.* are optional (and probably not used too heavily). I wonder if exposing query makes sense for this API since those are the two fields that you can query ?

I think exposing the query syntax here might be more confusing then the value it provides. Would it make sense to have this API ONLY return the watch IDs, so that a consumer would need go back to the /_watch/id endpoint ?

@@ -0,0 +1,25 @@
{
"watcher.list_watches":{
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to cc client's team for awareness

watcherMetadata = (Map<?, ?>) response.getWatches().get(1).getSource().getAsMap().get("metadata");
assertThat(watcherMetadata.get("key2"), equalTo(5));

request = new ListWatchesAction.Request(2, 2, null, List.of(new FieldSortBuilder("metadata.key1")), null);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also include a search_after here ? (or in a seperate test)

public static Watch createTestWatch(String watchName, Client client, HttpClient httpClient, EmailService emailService,
WatcherSearchTemplateService searchTemplateService, Logger logger) throws AddressException {
WatcherSearchTemplateService searchTemplateService, Logger logger) {
ActionThrottler actionThrottler = new ActionThrottler(Clock.systemUTC(), null, mock(XPackLicenseState.class));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this change for ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the reason this change is needed, is because the ListWatchesResponseTests creates watch instances for testing xcontent serialization and a couple more components are needed to make a watch xcontent serialization work without NPE's being thrown.

@martijnvg
Copy link
Copy Markdown
Member Author

The _id field is only a keyword, and the metadata.* are optional (and probably not used too heavily). I wonder if exposing query makes sense for this API since those are the two fields that you can query?

We don't know how widely metadata.* fields are being used and what kind of queries are used to retrieve watches via the search api. I don't think we should take this away from anyone that might be using this.

I think exposing the query syntax here might be more confusing then the value it provides. Would it make sense to have this API ONLY return the watch IDs, so that a consumer would need go back to the /_watch/id endpoint ?

How the query syntax behaves is the same whether users query the .watches index via the _search api or this new api. Today if you query the .watches index by fields that aren't indexed then either no hits are returned or an error (depending on the used query). I think just returning watch ids is pushing more complexity to the watcher ui and our users. Also the list watches api is a simple wrapper api without additional complexity. So I think the list watches api should return watches and keep the ability to query it.

@jakelandis
Copy link
Copy Markdown
Contributor

We don't know how widely metadata.* fields are being used and what kind of queries are used to retrieve watches via the search api.

Agree we don't know, but I think we can we can also agree that there is probably not a lot of searching happening given that the only 2 (logical) fields available to search, and one is a keyword and the other may not exist.

I don't think we should take this away from anyone that might be using this.

Assuming this is 8.0, we should look to subtract complexity for areas that are not really used and there are viable alternatives.

How the query syntax behaves is the same whether users query the .watches index via the _search api or this new api.

The syntax is the same, but there is subtle difference with the privilege model of _list_watches vs. _search and we don't expose aggregations.

Today if you query the .watches index by fields that aren't indexed then either no hits are returned or an error (depending on the used query)

Agree... however, I think this speaks to my point about it really being useful.

I think just returning watch ids is pushing more complexity to the watcher ui and our users.

I disagree here. This is a fairly common REST pattern (get IDs then go get additional info) and don't think it is that complex. I actually think using this API that supports search is more complex then a basic "list all of the ids". To list the watches I need to a) understand the API well enough to know about pagination or the default count returned (else I only get partial results) b) optionally understand the query syntax as well as the internal mappings (and that the only thing that i can search is metadata and IDs).

If we were to build this from scratch, would we expose the ability to search .watches by ID or metadata ? IMO, probably not. We would either not expose search, or index additional values. I don't think the .watches was ever really intended to be searched, hence the mappings.

If we go this path (only returning IDs from list), I do think we should extend the GET /_watcher/id to accept multiple id's.

@martijnvg
Copy link
Copy Markdown
Member Author

I disagree here. This is a fairly common REST pattern (get IDs then go get additional info) and don't think it is that complex. I actually think using this API that supports search is more complex then a basic "list all of the ids". To list the watches I need to a) understand the API well enough to know about pagination or the default count returned (else I only get partial results) b) optionally understand the query syntax as well as the internal mappings (and that the only thing that i can search is metadata and IDs).

The supported options are the same as in the search api, so no need to learn new syntax. Users that are querying the metadata fields are already aware of the mappings. I don't think we build this api for new users. New users should use Alerting instead.

If the list watch api would return just all the IDs of watches then we would still use the search api. The search api by default already includes the source of a watch. The api instead either doesn't request the _source or omits it, but then in a later point in time, any consumer of this API would then use the get watch api to get the watch source, which we could have returned in list watch response.

Also in the case that the list watch api returns all IDs then we would maybe need some form of pagination. I believe there are use cases were there is a watch per user model and in those cases there may be many watches.

If we were to build this from scratch, would we expose the ability to search .watches by ID or metadata ? IMO, probably not. We would either not expose search, or index additional values. I don't think the .watches was ever really intended to be searched, hence the mappings.

If I recall correctly, the metadata object field was added in order to select a subset of watches.

the only 2 (logical) fields available to search, and one is a keyword and the other may not exist.

This can be many fields defined by watch authors, since the metadata is an object field and many fields can exist under it.

If we go this path (only returning IDs from list), I do think we should extend the GET /_watcher/id to accept multiple id's.

I initially wanted to extend the get watch api, but the response currently doesn't allow returning multiple watches in backwards compatible way. We need api versioning in order to change the get watch api response.

@jakelandis
Copy link
Copy Markdown
Contributor

The supported options are the same as in the search api, so no need to learn new syntax.

My concern isn't so much about the syntax of the query, but rather needing to read the documentation for this API well enough to know _list_watches is really a (paginated) _search_watches but without full capabilities of _search and that you pretty limited to what you can actually search for.

If we changed the name to _watcher/_search (or something that does not include list) I might be in more favour of this change ? I think this might be the first case of where we are migrating people away from _search for a system index and want to consider the precedent we are setting. IMO the need to actually search .watches is a very limited use case, especially since the only thing you can really search is the metadata. I wonder if the search capability falls to the general exceptional case of being to query system indices, for which I believe there is (or will be) a "in case of emergency break glass" way to search these.

Also in the case that the list watch api returns all IDs then we would maybe need some form of pagination. I believe there are use cases were there is a watch per user model and in those cases there may be many watches.

I think returning a thousand IDs is still sane, and let the client do the pagination if necessary. If there are real cases of tens/hundreds of thousands watches (which is possible, but likely rare) we could introduce pagination as option, where the default is no pagination or a huge default value. Which IMO would be more inline with the semantics with list.

I initially wanted to extend the get watch api, but the response currently doesn't allow returning multiple watches in backwards compatible way.

I don't like it ... but ... we could in 7.x continue to return a single object for a single id (maintaining passivity), and an array of objects for multiple ids (new functionality). Then in 8.0 consolidate on a array objects and implement the compatible API the preserve the 7.x shape for single ids.

@jethr0null - Would you share your opinion ?

@martijnvg
Copy link
Copy Markdown
Member Author

If we changed the name to _watcher/_search (or something that does not include list) I might be in more favour of this change ?

That makes sense. Maybe /_watcher/_query?

If there are real cases of tens/hundreds of thousands watches (which is possible, but likely rare) we could introduce pagination as option

It is rare compared to all clusters that use Watcher, but there are cases with more then 10000 watches and even beyond 100000 watches or more, so I think we should take these clusters into account.

I don't like it ... but ... we could in 7.x continue to return a single object for a single id (maintaining passivity), and an array of objects for multiple ids (new functionality). Then in 8.0 consolidate on a array objects and implement the compatible API the preserve the 7.x shape for single ids.

For 7.x, perhaps we can switch to object array in the response if no watch id is provided? So when /_watcher/watch is invoked? Not sure... watch part indicates a single watch.

@cjcenizal
Copy link
Copy Markdown
Contributor

If I recall correctly, the metadata object field was added in order to select a subset of watches.

Not germane to this discussion, but the Watcher UI uses metadata to store a user-defined name value and also to store a type value. The type value is used to determine whether the watch can be edited with a user-friendly form or whether it's so complex that it can only be edited with a JSON editor.

@jakelandis
Copy link
Copy Markdown
Contributor

jakelandis commented Nov 30, 2020

Thanks CJ, I wasn't aware that the UI used this [1]. It appears that it simply read (as opposed to searched) for this use case ? However, this does provide a nice example of how it could be used for something non-trival as part of a larger system.

@martijnvg - Can we change the API to be /_watcher/_query ? (and I will +1 this change)

[1] example usage:

          "metadata" : {
            "name" : "test1",
            "watcherui" : {
              "trigger_interval_unit" : "h",
              "agg_type" : "count",
              "time_field" : "@timestamp",
              "trigger_interval_size" : 1,
              "term_size" : 5,
              "time_window_unit" : "m",
              "threshold_comparator" : ">",
              "term_field" : null,
              "index" : [
                "logstash-2020.11.30-000001"
              ],
              "time_window_size" : 5,
              "threshold" : 1000,
              "agg_field" : null
            },
            "xpack" : {
              "type" : "threshold"
            }
          }

@martijnvg martijnvg changed the title Add list watches api to retrieve multiple watches. Add query watches api to retrieve multiple watches. Dec 1, 2020
@martijnvg martijnvg requested a review from jakelandis December 1, 2020 08:52
Copy link
Copy Markdown
Contributor

@jakelandis jakelandis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks for the name change.

One minor point, however, I don't have a strong preference (only minor) would be for the path to _watcher/_query/watches I think this read a bit better and sets a bit cleaner precedence for custom queries against system indices.

@martijnvg martijnvg merged commit 3adb6d8 into elastic:master Dec 3, 2020
martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request Dec 3, 2020
This api supports pagination (from / size) and
querying and sorting by watch _id and watcher metadata.

This avoids using .watch index directly.
On a per watch basis the same information that the get watch api returns is returned,
except version.

Relates elastic#62501
martijnvg added a commit that referenced this pull request Dec 3, 2020
Backport #64582 to 7.x branch.

This api supports pagination (from / size) and
querying and sorting by watch _id and watcher metadata.

This avoids using .watch index directly.
On a per watch basis the same information that the get watch api returns is returned,
except version.

Relates #62501
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants