Describe the feature or problem you’d like to solve

Currently, when trying to use gh api with --paginate --slurp alongside --jq I get the following error:

the --slurp option is not supported with --jq or --template

I would have expected to be able to use these together, most likely getting a top-level array of all of the responses, that I could then process however I like with --jq syntax.

For example, using only --paginate but not --slurp:

⇒ gh api notifications --method GET -F participating=false -F all=true -F page=1 -F per_page=50 --paginate --jq 'length'

50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
14

Using --paginate --slurp with --jq:

⇒ gh api notifications --method GET -F participating=false -F all=true -F page=1 -F per_page=50 --paginate --slurp --jq 'length' | subl

the `--slurp` option is not supported with `--jq` or `--template`

Using --paginate --slurp and then needing to call out to external jq for further processing:

⇒ gh api notifications --method GET -F participating=false -F all=true -F page=1 -F per_page=50 --paginate --slurp | jq 'length'
24

⇒ gh api notifications --method GET -F participating=false -F all=true -F page=1 -F per_page=50 --paginate --slurp | jq 'map(length)'
[
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  14
]

For a more real-world example of what I was trying to achieve with --paginate and --jq; that I was then expecting --slurp would help me with (note that I am starting from page 20 here to reduce the length of the output for demonstration purposes):

⇒ gh api notifications --method GET -F participating=false -F all=true -F page=20 -F per_page=50 --paginate --jq '
  # Filter for notifications whose subject type is 'Release',
  # group them by repository, and output repo name and count sorted descending.
  [ .[] | select(.subject.type == "Release") ]
  | group_by(.repository.full_name)
  | map({ repo: .[0].repository.full_name, count: length })
  | sort_by(.count) | reverse
  #| map("\(.repo)  \(.count)")
  # | .[]
'

[{"count":3,"repo":"invoke-ai/InvokeAI"},{"count":2,"repo":"refined-github/refined-github"},{"count":2,"repo":"reduxjs/redux-toolkit"},{"count":2,"repo":"ollama/ollama"},{"count":2,"repo":"aws/aws-cdk"},{"count":1,"repo":"supabase/supabase"},{"count":1,"repo":"remix-run/react-router"},{"count":1,"repo":"obsidianmd/obsidian-releases"},{"count":1,"repo":"langflow-ai/langflow"},{"count":1,"repo":"facebook/react"},{"count":1,"repo":"OpenInterpreter/open-interpreter"}]
[{"count":6,"repo":"invoke-ai/InvokeAI"},{"count":3,"repo":"holochain/holochain"},{"count":2,"repo":"stripe/stripe-js"},{"count":2,"repo":"aws/aws-cdk"},{"count":1,"repo":"reduxjs/redux-toolkit"},{"count":1,"repo":"jehna/humanify"},{"count":1,"repo":"firebase/firebase-js-sdk"},{"count":1,"repo":"actions/checkout"},{"count":1,"repo":"Aider-AI/aider"}]
[{"count":4,"repo":"invoke-ai/InvokeAI"},{"count":3,"repo":"holochain/holochain"},{"count":1,"repo":"stripe/stripe-js"},{"count":1,"repo":"refined-github/refined-github"},{"count":1,"repo":"microsoft/autogen"},{"count":1,"repo":"firebase/firebase-js-sdk"},{"count":1,"repo":"aws/aws-cdk"},{"count":1,"repo":"actions/checkout"},{"count":1,"repo":"Readarr/Readarr"},{"count":1,"repo":"ProxymanApp/Proxyman"},{"count":1,"repo":"Aider-AI/aider"}]
[{"count":3,"repo":"invoke-ai/InvokeAI"},{"count":2,"repo":"ollama/ollama"},{"count":2,"repo":"aws/aws-cdk"},{"count":1,"repo":"refined-github/refined-github"},{"count":1,"repo":"holochain/holochain"},{"count":1,"repo":"firebase/firebase-js-sdk"},{"count":1,"repo":"Whisky-App/Whisky"},{"count":1,"repo":"JedWatson/react-select"},{"count":1,"repo":"Heroic-Games-Launcher/HeroicGamesLauncher"},{"count":1,"repo":"Aider-AI/aider"}]
[{"count":1,"repo":"supabase/supabase"},{"count":1,"repo":"stripe/stripe-js"},{"count":1,"repo":"holochain/holochain"}]

Essentially, the --paginate data seems to be processed individually by --jq; so the output is sort of reminiscent of JSON Lines:

• https://jsonlines.org/

Because each page of the --paginate'd data is treated separately by --jq, when I try and aggregate/count, it doesn't really work, as in this case I get ~24 pages of counts by repo, rather than 1 full count by repo.

By my reading of the vast history of related issues/PR's (linked below), I believe this was kept as there was a desire to not introduce potentially breaking changes before v3 (even if the likelihood of this having any real world user impact was probably low, but somewhat unquantifiable). I believe this was the reason that --slurp was introduced, as a potentially temporary addon so that --paginate's old functionality remained, but we could opt in to a single JSON return value by using --slurp.

Proposed solution

When I was first reading through #8620 (after eventually getting to it through all the past related context linked below), I was expecting that the whole point of that PR was so I could use --paginate --slurp with --jq to avoid needing to rely on an external program (eg. jq); as otherwise, I believe we could have just always relied on an external jq --slurp with the old --paginate implementation.

I would really love to be able to use --paginate, and then process that with --jq; and have it all treated as a 'single pass', rather than the JSON Lines style processing we currently get. My understanding of --slurp is that that would be ideal for this; but we can't use that with --jq currently.

Additional context

I spent a bunch of time reading through the long history of related issues.. (with a LOT of effort put in by @heaths, along with a bunch of others along the way) and while I'm not necessarily deeply wiser about the full spectrum of implementation choices explored and iterated through; here is my attempted shortlist of related issues/PR's in hopefully roughly relevant historical order:

• Output a single JSON document from api --paginate #1268

  • Closed as completed in Merge JSON responses from gh api #8620

• Merge JSON responses from gh api #5652

  • Closing in lieu of Merge JSON responses from gh api #8620

• api: output a single JSON array in REST pagination mode #7190
• Support merging JSON arrays, objects go-gh#148

  • Closing per discussion above and in Merge JSON responses from gh api #8620

• Merge JSON responses from gh api #8620
  • Merge JSON responses from gh api #8620 (comment)

    • Like before, it will not be used for --jq or --template as this is implemented as an io.Writer.

      I suppose another way would be to implement this as a reader similar to the existing REST merge, but then you'd have more problems sending incomplete JSON documents to jsoncolor. The only advantage is --template and --jq would get the same JSON array container representation if --slurp were provided. Without a complete overhaul of how JSON is effectively merged, there are pros and cons either way. This way, at least, seems fairly straight forward.

• Allow RESTClient and GQLClient to support automatic pagination go-gh#23

Tangentially related, but probably not relevant (including because it recently touched a similar area, so not sure if there was a regression, but based on above context, I think probably isn't relevant):

• Incorrect error shown (the --slurp option is not supported with --jq or --template) when passing --jq + --template without --slurp #10260
  • [gh api] Fix mutual exclusion messages of --slurp flag #10332

Similarly, I'm not sure if this is only tangentially related or not.. but it feels in a similar space of things, so linking in case the additional context is helpful:

• Support getting both JSON and template output #8415

Tangentially Related

There have been a number of other json/jq Quality of Life issues I've opened recently; and some awesome PRs associated with some of them, made by @heaths . I'm not sure of the deeper specifics as to whether any of them would crossover with this at an implementation level; but from a user perspective, it's kind of weird/confusing feeling when different CLI commands seem to handle json/jq differently.

• Allow --jq and --template to be used together in gh api #10261
  • Support getting both JSON and template output #8415
    • Support filtering before applying template go-gh#177
    • Process --jq before --template #10273
• Add custom formatting functions to --jq similar to those added to --template #10262
  • Add useful template functions to jq filters go-gh#178
  • Support useful template functions and modules in jq filters #10275
• Allow passing additional arguments through to --jq script, just like we can with jq --arg #10263

Originally posted by @0xdevalias in #10385