Skip to content

chore(docs): Add more godoc to important APIs #118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kodiakhq merged 11 commits into from
Sep 18, 2022
Merged

chore(docs): Add more godoc to important APIs #118

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
8282264
chore(docs): Add more godoc to important APIs
yevgenypats Sep 16, 2022
30f0353
Update codegen/doc.go
yevgenypats Sep 18, 2022
9a79683
Update schema/doc.go
yevgenypats Sep 18, 2022
31e9bc1
Update specs/source.go
yevgenypats Sep 18, 2022
8aca4c3
Update specs/source.go
yevgenypats Sep 18, 2022
12eacd3
Update specs/source.go
yevgenypats Sep 18, 2022
515e2a3
Update specs/source.go
yevgenypats Sep 18, 2022
1d0a9e4
Update specs/source.go
yevgenypats Sep 18, 2022
873a584
Update plugins/docs.go
yevgenypats Sep 18, 2022
ac3a859
Update serve/doc.go
yevgenypats Sep 18, 2022
9aa796d
Merge branch 'main' into feat/more_docs
kodiakhq[bot] Sep 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 7 additions & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,16 @@
CloudQuery Plugin SDK ![License](https://img.shields.io/github/license/cloudquery/cloudquery?style=flat-square)
=======================
# CloudQuery Plugin SDK

This SDK enables building CloudQuery plugins which allows CloudQuery's users to extract/load/transform existing and popular service providers as well as custom in-house solutions into a SQL Database.
CloudQuery SDK enables building CloudQuery source and destination plugins.

CloudQuery itself is a tool that transforms your cloud infrastructure into queryable SQL for easy monitoring, governance and security. You can find more about CloudQuery on its website and its GitHub repository.
Source plugins allows developers to extract information from third party APIs and enjoying built-in transformations, concurrency, logging, testing and database agnostic support via destination plugins.

Destinations plugins allows writing the data from any of the source plugins to an additional database, message queue, storage or any other destination without recompiling any of the source plugins.

## Getting Started & Documentation

* Homepage: https://cloudquery.io
* Releases: https://github.com/cloudquery/cloudquery/releases
* Documentation: https://docs.cloudquery.io
* Database Configuration: https://docs.cloudquery.io/database-configuration
- Homepage: https://cloudquery.io
- Releases: https://github.com/cloudquery/cloudquery/releases
- Documentation: https://docs.cloudquery.io

## Supported plugins

Expand Down
3 changes: 3 additions & 0 deletions clients/doc.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
// package clients is a wrapper around grpc clients so clients can work
// with non protobuf structs and handle unmarshaling
package clients
3 changes: 1 addition & 2 deletions clients/source.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
// package clients is a wrapper around grpc clients so clients can work
// with non protobuf structs and handle unmarshaling
package clients

import (
Expand All @@ -14,6 +12,7 @@ import (
"google.golang.org/grpc"
)

// SourceClient
type SourceClient struct {
pbClient pb.SourceClient
}
Expand Down
2 changes: 1 addition & 1 deletion codegen/doc.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
// codgen helps autogenerate cloudquery plugins configured by definition
// Package codgen helps autogenerate tables for CloudQuery source plugins from Go structs (of relevant SDKs)
package codegen
1 change: 1 addition & 0 deletions codegen/golang.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ func (t *TableDefinition) addColumnFromField(field reflect.StructField, parentFi
t.Columns = append(t.Columns, column)
}

// NewTableFromStruct creates a new TableDefinition from a struct by inspecting its fields
func NewTableFromStruct(name string, obj interface{}, opts ...TableOptions) (*TableDefinition, error) {
t := &TableDefinition{
Name: name,
Expand Down
2 changes: 2 additions & 0 deletions plugins/docs.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
// Package plugins defines APIs for source and destination plugins
package plugins
9 changes: 8 additions & 1 deletion plugins/source.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ func addInternalColumns(tables []*schema.Table) {
}
}

// NewSourcePlugin returns a new plugin with a given name, version, tables, newExecutionClient
// and additional options.
func NewSourcePlugin(name string, version string, tables []*schema.Table, newExecutionClient SourceNewExecutionClientFunc, opts ...SourceOption) *SourcePlugin {
p := SourcePlugin{
name: name,
Expand Down Expand Up @@ -91,27 +93,32 @@ func (p *SourcePlugin) validate() error {
return nil
}

// Tables returns all supported tables by this source plugin
func (p *SourcePlugin) Tables() schema.Tables {
return p.tables
}

// ExampleConfig returns an example configuration for this source plugin
Copy link
Copy Markdown
Member

@erezrokah erezrokah Sep 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// ExampleConfig returns an example configuration for this source plugin
// Returns example configuration specific for this source plugin

func (p *SourcePlugin) ExampleConfig() string {
return p.exampleConfig
}

// Name return the name of this plugin
Copy link
Copy Markdown
Member

@erezrokah erezrokah Sep 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Name return the name of this plugin
// Returns a human friendly name for the plugin

func (p *SourcePlugin) Name() string {
return p.name
}

// Version returns the version of this plugin
Copy link
Copy Markdown
Member

@erezrokah erezrokah Sep 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Version returns the version of this plugin
// Returns the semantic valid version of the plugin

func (p *SourcePlugin) Version() string {
return p.version
}

// SetLogger sets the logger for this plugin which will be used in Sync and all other function calls.
func (p *SourcePlugin) SetLogger(log zerolog.Logger) {
p.logger = log
}

// Sync data from source to the given channel
// Sync is syncing data from the requested tables in spec to the given channel
Copy link
Copy Markdown
Member

@erezrokah erezrokah Sep 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Sync is syncing data from the requested tables in spec to the given channel
// Sync retrieves data from the requested tables in the plugin's specification and sends the results via the provider `res` channel

func (p *SourcePlugin) Sync(ctx context.Context, spec specs.Source, res chan<- *schema.Resource) error {
c, err := p.newExecutionClient(ctx, p.logger, spec)
if err != nil {
Expand Down
2 changes: 2 additions & 0 deletions schema/doc.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
// Package schema defines types supported by tables in source plugins
package schema
1 change: 1 addition & 0 deletions serve/doc.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
// Package serve defines APIs to serve (invoke) source and destination plugins
package serve

import (
Expand Down
3 changes: 3 additions & 0 deletions specs/doc.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
// Package specs specs for source and destination plugins including
Copy link
Copy Markdown
Member

@erezrokah erezrokah Sep 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand what message this comment is trying to convey

// parsers and readers.
package specs
32 changes: 22 additions & 10 deletions specs/source.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,30 @@ import (
"github.com/xeipuuv/gojsonschema"
)

// Source is the shared configuration for all source plugins
// Source is the spec for a source plugin
Copy link
Copy Markdown
Member

@erezrokah erezrokah Sep 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Source is the spec for a source plugin
// Source defines the specification for a source plugin

type Source struct {
Name string `json:"name,omitempty"`
// Name of the source plugin to use
Copy link
Copy Markdown
Member

@erezrokah erezrokah Sep 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Name of the source plugin to use
// Human friendly name of the plugin

Name string `json:"name,omitempty"`
// Version of the source plugin to use
Copy link
Copy Markdown
Member

@erezrokah erezrokah Sep 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Version of the source plugin to use
// Valid semantic version of the plugin

Version string `json:"version,omitempty"`
// Path is the path in the registry
// Path is the canonical path to the source plugin in a given registry
// For example:
// in github the path will be: org/repo
Copy link
Copy Markdown
Member

@erezrokah erezrokah Sep 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// in github the path will be: org/repo
// For the GitHub registry the path will be: org/repo

// For the local registry the path will be the path to the binary: ./path/to/binary
// For the gRPC registry the path will be the address of the gRPC server: host:port
Path string `json:"path,omitempty"`
// Registry can be github,local,grpc. Might support things like https in the future.
Registry Registry `json:"registry,omitempty"`
MaxGoRoutines uint64 `json:"max_goroutines,omitempty"`
Tables []string `json:"tables,omitempty"`
SkipTables []string `json:"skip_tables,omitempty"`
Destinations []string `json:"destinations,omitempty"`
Spec interface{} `json:"spec,omitempty"`
// Registry can be github,local,grpc.
Registry Registry `json:"registry,omitempty"`
MaxGoRoutines uint64 `json:"max_goroutines,omitempty"`
// Tables to sync from the source plugin
Tables []string `json:"tables,omitempty"`
// SkipTables defines tables to skip when syncing data. Useful if a glob pattern is used in Tables
SkipTables []string `json:"skip_tables,omitempty"`
// Destinations are the names of destination plugins to send sync data to
Destinations []string `json:"destinations,omitempty"`
// Spec defines plugin specific configuration
// This is different in every source plugin.
Spec interface{} `json:"spec,omitempty"`
}

func (s *Source) SetDefaults() {
Expand All @@ -37,6 +48,7 @@ func (s *Source) SetDefaults() {
}
}

// UnmarshalSpec unmarshals the internal spec into the given interface
func (s *Source) UnmarshalSpec(out interface{}) error {
b, err := json.Marshal(s.Spec)
if err != nil {
Expand Down