Skip to content

feat: Add TableSet type#475

Closed
hermanschaaf wants to merge 4 commits intomainfrom
refactor
Closed

feat: Add TableSet type#475
hermanschaaf wants to merge 4 commits intomainfrom
refactor

Conversation

@hermanschaaf
Copy link
Copy Markdown
Contributor

This implements the set idea from #470

It doesn't change any behavior, and doesn't fix the bug in TableForSpec. It's just for readability: listAndValidateTables now returns a set type which only supports a very limited set of operations. This set type is passed to syncDfs rather than a flattened list. syncDfs and its related functions must reference p.tables to traverse the tables, but can now check selectedTables.Contains(table.Name) to see whether a given table should be included or skipped. I think this makes the code quite a bit simpler (e.g. no more checks for whether a table is top-level or not) and less error-prone.

// listAndValidateTables returns a flattened list - we only want to return
// the top-level tables from this function.
// listAndValidateTables returns a set of all tables - we only want to return
// the top-level tables from the set in this function.
Copy link
Copy Markdown
Contributor Author

@hermanschaaf hermanschaaf Dec 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically I think this behavior should change (this function should return all tables, not just top-level ones), but I'm not changing it in this PR.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Dec 8, 2022

⏱️ Benchmark results

Comparing with cdcef5d

  • DefaultConcurrency-2 resources/s: 11,662 ⬇️ 3.10% decrease vs. cdcef5d
  • Glob-2 ns/op: 164.5 ⬇️ 26.63% decrease vs. cdcef5d
  • TablesWithChildrenDefaultConcurrency-2 resources/s: 30,227 ⬆️ 11.60% increase vs. cdcef5d
  • BufferedScanner-2 ns/op: 10.11 ⬇️ 26.61% decrease vs. cdcef5d
  • LogReader-2 ns/op: 30.72 ⬇️ 30.70% decrease vs. cdcef5d

@yevgenypats
Copy link
Copy Markdown
Contributor

Closing in favour of #488

kodiakhq bot pushed a commit that referenced this pull request Dec 12, 2022
This simplifies our glob filtering and makes sure Tables are always stored in a tree like structured apart from when returned from `FlattenTables`. This solves a lot of potential bugs and places where we need to do things like `if parent != nil`.

I think this also closes #475 as there is no need for another data structure.

The reasoning for this is that our tables are tree like structured and it's easier imo to always keep it like this everywhere across the code and not have multiple data structures with similar methods and so on. This required a small update to our filtering logic which just works as a DFS and in-place filtering (for that to work I had to add a copy method which I think can be  useful for other things in the future).
@candiduslynx candiduslynx deleted the refactor branch June 6, 2024 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants