Describe the enhancement: The httpjson filebeat input only works if the API contains all information in the JSON. However there are a lot of APIs where pagination is not part of the request JSON (because there may be none), but the pagination information is stored in HTTP query parameters. Those APIs should be support as well. This also means that one needs to have the ability of to keep state on the beats side like counting up an integer, as many APIs do not offer that as part of their response.
Describe a specific use case for the enhancement or feature: APIs like the GitHub API use HTTP query parameters for pagination, i.e. per_page and page for the current page number. See https://docs.github.com/en/rest/reference/orgs#list-organization-members
Describe the enhancement: The httpjson filebeat input only works if the API contains all information in the JSON. However there are a lot of APIs where pagination is not part of the request JSON (because there may be none), but the pagination information is stored in HTTP query parameters. Those APIs should be support as well. This also means that one needs to have the ability of to keep state on the beats side like counting up an integer, as many APIs do not offer that as part of their response.
Describe a specific use case for the enhancement or feature: APIs like the GitHub API use HTTP query parameters for pagination, i.e.
per_pageandpagefor the current page number. See https://docs.github.com/en/rest/reference/orgs#list-organization-members