Skip to content

feat: Separate file parts#154

Merged
kodiakhq[bot] merged 4 commits intomainfrom
feat/filetypes-separate-parts
Jun 7, 2023
Merged

feat: Separate file parts#154
kodiakhq[bot] merged 4 commits intomainfrom
feat/filetypes-separate-parts

Conversation

@disq
Copy link
Member

@disq disq commented May 9, 2023

This is 100% compatible with the 'old ways' because it still implements WriteTableBatchFile with unchanged signature.

Read is not refactored due to the interface differences between parquet and the other types.

@github-actions github-actions bot added the feat label May 9, 2023
@disq disq requested a review from hermanschaaf May 10, 2023 08:36
@github-actions github-actions bot added feat and removed feat labels May 10, 2023
type Client struct {
spec *FileSpec

types.FileType
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is used for all interactions with the underlying writer. For read tests, separate Clients are used which are still below.

csv/write.go Outdated

var _ types.Handle = (*Handle)(nil)

func (cl *Client) WriteHeader(w io.Writer, s *arrow.Schema) (types.Handle, error) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this particular format/package, the csv header is written with the first call to .Write (and not in this fn) so the name might be misleading. Maybe rename the interface method to 'Begin' etc?

@disq disq marked this pull request as ready for review May 10, 2023 08:42
@disq disq marked this pull request as draft May 11, 2023 10:33
@github-actions github-actions bot added feat and removed feat labels May 11, 2023
@disq disq force-pushed the feat/filetypes-separate-parts branch from 5b45f8e to 8260f76 Compare May 11, 2023 14:23
@disq disq marked this pull request as ready for review May 12, 2023 08:55
@github-actions github-actions bot added feat and removed feat labels May 12, 2023
@disq disq requested a review from yevgenypats May 15, 2023 09:11
@github-actions github-actions bot added feat and removed feat labels May 15, 2023
@disq disq force-pushed the feat/filetypes-separate-parts branch 2 times, most recently from efd328f to c274bcb Compare May 25, 2023 07:43
@disq disq force-pushed the feat/filetypes-separate-parts branch from 7a1d607 to 5b43e44 Compare June 1, 2023 09:28
@disq disq requested a review from candiduslynx June 6, 2023 11:20
@disq disq added the automerge Add to automerge PRs once requirements are met label Jun 7, 2023
@kodiakhq kodiakhq bot merged commit 3798fab into main Jun 7, 2023
@kodiakhq kodiakhq bot deleted the feat/filetypes-separate-parts branch June 7, 2023 18:04
kodiakhq bot pushed a commit that referenced this pull request Jun 14, 2023
🤖 I have created a release *beep* *boop*
---


## [3.2.0](v3.1.6...v3.2.0) (2023-06-14)


### Features

* Separate file parts ([#154](#154)) ([3798fab](3798fab))


### Bug Fixes

* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 4d76231 ([#198](#198)) ([aafa961](aafa961))
* **deps:** Update module github.com/cloudquery/plugin-sdk/v3 to v3.10.5 ([#201](#201)) ([0c8519c](0c8519c))
* **deps:** Update module github.com/cloudquery/plugin-sdk/v3 to v3.10.6 ([#202](#202)) ([a94a865](a94a865))
* Use buffered writing mode in parquet ([#203](#203)) ([5ad67f3](5ad67f3))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge Add to automerge PRs once requirements are met feat

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants