Skip to content

feat: Add GH Workflow support#3195

Merged
kodiakhq[bot] merged 10 commits intocloudquery:mainfrom
bbernays:add-workflow-support
Dec 12, 2022
Merged

feat: Add GH Workflow support#3195
kodiakhq[bot] merged 10 commits intocloudquery:mainfrom
bbernays:add-workflow-support

Conversation

@bbernays
Copy link
Copy Markdown
Collaborator

@bbernays bbernays commented Oct 25, 2022

Summary

Add support Github Actions

To do:

  • Add unit test
  • Add Support For GetWorkflowFile

@bbernays bbernays requested review from a team and candiduslynx and removed request for a team October 25, 2022 14:08
@cq-bot cq-bot added the github label Oct 25, 2022
@bbernays bbernays requested a review from yevgenypats October 25, 2022 14:08
Comment on lines +13 to +39
opts := &github.RepositoryListByOrgOptions{ListOptions: github.ListOptions{PerPage: 100}}
for {
repos, resp, err := c.Github.Repositories.ListByOrg(ctx, c.Org, opts)
if err != nil {
return err
}
for _, repo := range repos {
actionOpts := &github.ListOptions{PerPage: 100}
for {
workflows, resp, err := c.Github.Actions.ListWorkflows(ctx, *repo.Owner.Login, *repo.Name, actionOpts)
if err != nil {
return err
}
res <- workflows.Workflows
opts.Page = resp.NextPage
if opts.Page == resp.LastPage {
break
}
}
}
opts.Page = resp.NextPage
if opts.Page == resp.LastPage {
break
}
}
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not optimized as the only multiplexer that is available is an org multiplexer

Copy link
Copy Markdown
Member

@erezrokah erezrokah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bbernays this works pending a way to multiplex repositories (unrelated to this PR).

Approved pending a mock test for it. Is GetWorkflowFile required? Looking the GitHub API we get the same data as the list operation:
https://docs.github.com/en/rest/actions/workflows#get-a-workflow

@erezrokah
Copy link
Copy Markdown
Member

@bbernays Should we bring this PR up to speed?

@github-actions
Copy link
Copy Markdown

This PR has the following changes to source plugin(s) tables:

  • Table github_workflows was added

@erezrokah
Copy link
Copy Markdown
Member

erezrokah commented Dec 12, 2022

So I brought this PR up to speed in c7d7029 and cbfe918.

cbfe918 also changes the contents column so:

  1. It's a string and not a JSON. We can't guarantee that the workflow file is a valid YAML (it can have syntax errors, also GitHub has a custom parser for it so it's does not follow the spec). Better to keep the data raw
  2. Uses HTMLURL as that also has the branch name, which will give us a better representation of the file's content
  3. If we fail to set get the content, set the column value to nil instead of failing

@erezrokah erezrokah added the automerge Automatically merge once required checks pass label Dec 12, 2022
@kodiakhq kodiakhq bot merged commit 098ac50 into cloudquery:main Dec 12, 2022
kodiakhq bot pushed a commit that referenced this pull request Dec 13, 2022
🤖 I have created a release *beep* *boop*
---


## [1.4.0](plugins-source-github-v1.3.0...plugins-source-github-v1.4.0) (2022-12-13)


### Features

* Add GH Workflow support ([#3195](#3195)) ([098ac50](098ac50))


### Bug Fixes

* **deps:** Update module github.com/cloudquery/plugin-sdk to v1.11.1 ([#5458](#5458)) ([58b7432](58b7432))
* **deps:** Update module github.com/cloudquery/plugin-sdk to v1.11.2 ([#5497](#5497)) ([c1876cf](c1876cf))
* **deps:** Update module github.com/cloudquery/plugin-sdk to v1.12.0 ([#5539](#5539)) ([fb71293](fb71293))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
@bbernays bbernays deleted the add-workflow-support branch December 13, 2022 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge Automatically merge once required checks pass

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants