Merging unitycatalog-ui repo into unitycatalog (main) repo #349

dennyglee · 2024-08-13T15:59:08Z

dennyglee
Aug 13, 2024
Maintainer

From the last few weeks of discussions and development, it has become apparent that the UI depends on the main repo YAML. For ease of development, debugging, and deployment, we have been discussing merging this before deploying the solution around npm.

Note this is an executive summary from the discussions on the Unity Catalog Slack unitycatalog-ui Slack channel and our weekly unitycatalog-ui contributors meetings.

If there are any concerns, please chime in here as we would like to merge the repos on Thursday, August 15th. We will archive the original unitycatalog-ui repo and refer to this discussion.

dennyglee · 2024-08-13T16:00:13Z

dennyglee
Aug 13, 2024
Maintainer Author

@ognis1205 asked a great question during today's unitycatalog-ui contributors' meeting:

I don't have particularly strong opinions about the integration of the unitycatalog-ui into the unitycatalog repository, but when it comes to managing the OpenAPI specifications and the associated client implementations, I think it might be a good idea to set up a repository dedicated solely to managing the OpenAPI for Unity Catalog. Additionally, using CI/CD-based package management with GitHub Actions could be a viable approach. The idea would be to publish the corresponding version of the client as a Maven Artifact for JVM, a PyPi package for Python, or an npm package for TypeScript, at the time of release. This approach might be beneficial, although I believe it's necessary to have discussions beforehand about technical aspects and operational risks. This could help reduce the tasks managed within the unitycatalog repository, make API version management between the server and clients easier regardless of language, and potentially make it easier for various ecosystems to reuse the APIs. It's just an idea, though.

2 replies

dennyglee Aug 13, 2024
Maintainer Author

Hey @ognis1205, per today's meeting, you bring up a great point and in the long-term, I think this approach makes a lot of sense. The concern is that we're still making a lot of changes to the YAML. The purpose of this change is to simplify the development so we don't have to cross repos for development or subsequent docker builds. But once we stabilize the YAML/API, we should consider this approach as there are some serious advantages to this approach.

ognis1205 Aug 13, 2024

Thank you for reposting my question @dennyglee !

dennyglee · 2024-08-16T15:53:34Z

dennyglee
Aug 16, 2024
Maintainer Author

Thanks for the various side conversations as well! Looks like there are not any concerns with this approach - we will work on merging this repos. Thanks!

0 replies

dennyglee · 2024-08-22T15:45:58Z

dennyglee
Aug 22, 2024
Maintainer Author

Quick update, here is our current plan to merge the unitycatalog-ui repo into the unitycatalog repo to simplify UI development and debuggability. The basic steps are:

Resolve any open unitycatalog-ui PRs and issues by September 6th, 2024 5pm PT
Coalesce all unitycatalog-ui branches into main
Merge unitycatalog-ui main branch into unitycatalog repo ui-main to keep all history
Restructure the unitycatalog repo, main branch to add the ui folder (additional details below)
Verify everything looks good
Copy any remaining unitycatalog-ui issues over to unitycatalog
Archive unitycatalog-ui repo and update the README.md to note the merge (point to this discussion)

Are there any concerns with this approach? Some additional details if you would like to dive into this.

Additional Details

The restructured hierarchy of the main repo would be

| unitycatalog
|- api
|- ...
|- tests
|- ui (new)
   |- public
   |- src
   |- READMNE.md
   |- package.json
   |- ...

Note, the following unitycatalog-ui files would be removed after repo transfer

CONTRIBUTING.md
CODE_OF_CONDUCT.md
LICENSE.txt

9 replies

dennyglee Aug 22, 2024
Maintainer Author

The project directory structure seems fine from the perspective of OpenAPI code generation. However, one thing that concerns me is:

Resolve any open unitycatalog-ui PRs and issues by September 6th, 2024, 5 PM PT.

Copy any remaining unitycatalog-ui issues over to unitycatalog.

These two points seem to contradict each other a little. For example, is my understanding correct that my issue can be carried over after the merge? If that's the case, I have no objections to this plan.

Sorry for not clarifying @ognis1205 - what I meant by "copy any remaining issues" is that we create a new issue within the unitycatalog (main) repo for any remaining open issues in the unitycatalog-ui. The details will still be there but the idea is that any new issues would be created in main. WDYT @ognis1205 ?

ognis1205 Aug 23, 2024

@dennyglee

Thank you for elaborating the plan.

what I meant by "copy any remaining issues" is that we create a new issue within the unitycatalog (main) repo for any remaining open issues in the unitycatalog-ui.

So that means we don't need to resolve all the current PRs/issues before merging, correct? My issue depends on the all.yaml in the main repository, so I think it would be more efficient to work on it in the main repository after the merge, rather than copying the current all.yaml to the unitycatalog-ui and working there. Therefore, I'm planning to work in the main repository after the merge, but what does everyone think?

cc: @jamieknight-db @creechy

jamieknight-db Aug 23, 2024
Collaborator

@ognis1205 agreed that this issue will be easier once the repositories are merged, let's plan for that. 👍

ognis1205 Aug 24, 2024

@jamieknight-db

Thank you for the advice. I will try to proceed in that direction 👍

dennyglee Aug 27, 2024
Maintainer Author

Thanks @ognis1205 - based on this, we will see if we can merge the repos a little earlier so that way it can unblock downstream development like yours. Let's discuss during tomorrow's review and plan accordingly, eh?!

ognis1205 · 2024-08-28T05:54:47Z

ognis1205
Aug 28, 2024

FYI:

Many of you may already know this, but it seems that there's a movement to (re)define UC OSS's REST API using protocol buffers in mlflow as well.

0 replies

dennyglee · 2024-08-28T15:43:19Z

dennyglee
Aug 28, 2024
Maintainer Author

Based on today's UI contributor meeting, we will pause all current PRs so we can streamline the integrations. To do this, our current plan is:

@jamieknight-db will be opening the login UI PR as part of the feat/login-ui branch
- Due to backend (BE) integration dependencies, we will hide the login UI button/page
- The BE integrations will be completed in the main repo
Once the above PR is merged, we will merge the unitycatalog-ui repo into the main repo as there will be only one branch remaining

Key point: current plan is to merge of the unitycatalog-ui repo (main branch) into the unitycatalog repo (ui branch)

0 replies

dennyglee · 2024-09-02T22:55:26Z

dennyglee
Sep 2, 2024
Maintainer Author

Per the last conversations, we have

Merged the unitycatalog-ui/main into the unitycatalog/ui
Updated the unitycatalog-ui README.md to reflect the archival process
Upon debug/review, we will archive the unitycatalog-ui repo on September 5th, 2024.

1 reply

dennyglee Sep 4, 2024
Maintainer Author

Quick update, the restructuring of the ui into the /ui folder has been completed and you can find it in PR #408

ognis1205 · 2024-09-03T02:38:36Z

ognis1205
Sep 3, 2024

@everettVT

Sorry for the sudden mention. The UI repository has been merged into the main repository, so I’m planning to work on the issue we discussed earlier. You seemed interested in this, so I’d like to hear your thoughts during the review. Would you be interested?

4 replies

everettVT Sep 3, 2024

Absolutely. Happy to.

ognis1205 Sep 4, 2024

Thanks @everettVT !

My working repository is:

~~https://github.com/ognis1205/unitycatalog/tree/feat/use-openapi-specification~~

Once the PR is filed, I will ping you as a reviewer. If you haven't joined the Slack channel yet, I suggest you join the thread as well:

https://www.unitycatalog.io/

everettVT Sep 5, 2024

Hey @ognis1205,

I launched your working branch and it built and with no problems. I was able to create a new catalog, but not new tables. I am looking to see if its because my UC Server is running on localhost:3001 (since I'm running another process on 3000), but since I was able to create a catalog, I'm not sure that would really be it.

I am able to visualize the tables located in the default "unity" catalog, but nothing renders following volume or function selection.

Happy to hop on a call or something if this is not expected behavior. Let me know.

ognis1205 Sep 6, 2024

Sorry @everettVT,

The merge of the UI repository is currently suspended due to certain circumstances, so I haven't been working on my branch (meaning the UI implementation hasn't changed). Once the merge is complete and the work is done, I'll ping you again, but in the meantime, I'll check if I can reproduce the issue you're mentioning.

FYI:
There is a dedicated Slack channel for the UI, and depending on the situation, I think it would be worth considering creating an issue as well. As for the issue, I believe it will be created in the main repository moving forward. cc: @dennyglee @jamieknight-db

dennyglee · 2024-09-06T13:55:35Z

dennyglee
Sep 6, 2024
Maintainer Author

Okay, all - thanks to @rtyler and PR #421 (which was a fix to my previous blunder so thanks again!), the UI repo has been merged. Thanks!

3 replies

JuanPabloDiaz Sep 6, 2024

Thank you, everything looks good! the contribution history is there.

ognis1205 Sep 6, 2024

Thank you, all!

Do we need GitHub issue labels like java, typescript, (possibly python), or uiux?

dennyglee Sep 6, 2024
Maintainer Author

Very possible, since we have merged the UI with the main repo, definitely agree on the idea of uiux. Main concern I have is that if we add languages, we're adding a lot of extra labels. WDYT?

dennyglee · 2024-09-06T23:47:16Z

dennyglee
Sep 6, 2024
Maintainer Author

Thanks for you patience - as of September 6th, 2024 we have archived the unitycatalog-ui repo. As noted earlier in this discussion, all code can be found in the unitycatalog (main) repo, specifically /ui folder.

0 replies

JuanPabloDiaz · 2024-09-09T17:05:57Z

JuanPabloDiaz
Sep 9, 2024

Hello, I was about to open an issue but realized it's unnecessary. Now that the UI has been merged into the main repo, the warning message can be removed and perhaps add additional info in the UI readme.

1 reply

dennyglee Sep 10, 2024
Maintainer Author

That's a good call @JuanPabloDiaz - any chance you'd be up for creating an issue and/or PR to fix the UI message? Perhaps we can leverage/update the Quick Start > UI section?

ognis1205 · 2024-09-10T17:08:21Z

ognis1205
Sep 10, 2024

It seems better to create a new discussion to talk about the UI project in general.

2 replies

JuanPabloDiaz Sep 10, 2024

Good idea!

dennyglee Sep 10, 2024
Maintainer Author

Good idea! I just started this discussion so why don't we dive into it there?

If there are no concerns, I'll close this discussion next week to help facilitate the new discussion.

Thanks!

Merging unitycatalog-ui repo into unitycatalog (main) repo #349

Uh oh!

dennyglee Aug 13, 2024 Maintainer

Replies: 11 comments · 22 replies

Uh oh!

dennyglee Aug 13, 2024 Maintainer Author

Uh oh!

dennyglee Aug 13, 2024 Maintainer Author

Uh oh!

Uh oh!

dennyglee Aug 16, 2024 Maintainer Author

Uh oh!

dennyglee Aug 22, 2024 Maintainer Author

Additional Details

Uh oh!

dennyglee Aug 22, 2024 Maintainer Author

Uh oh!

Uh oh!

jamieknight-db Aug 23, 2024 Collaborator

Uh oh!

Uh oh!

dennyglee Aug 27, 2024 Maintainer Author

Uh oh!

Uh oh!

dennyglee Aug 28, 2024 Maintainer Author

Uh oh!

dennyglee Sep 2, 2024 Maintainer Author

Uh oh!

dennyglee Sep 4, 2024 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dennyglee Sep 6, 2024 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

dennyglee Sep 6, 2024 Maintainer Author

Uh oh!

dennyglee Sep 6, 2024 Maintainer Author

Uh oh!

Uh oh!

dennyglee Sep 10, 2024 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dennyglee Sep 10, 2024 Maintainer Author

dennyglee
Aug 13, 2024
Maintainer

Replies: 11 comments 22 replies

dennyglee
Aug 13, 2024
Maintainer Author

dennyglee Aug 13, 2024
Maintainer Author

dennyglee
Aug 16, 2024
Maintainer Author

dennyglee
Aug 22, 2024
Maintainer Author

dennyglee Aug 22, 2024
Maintainer Author

jamieknight-db Aug 23, 2024
Collaborator

dennyglee Aug 27, 2024
Maintainer Author

dennyglee
Aug 28, 2024
Maintainer Author

dennyglee
Sep 2, 2024
Maintainer Author

dennyglee Sep 4, 2024
Maintainer Author

dennyglee
Sep 6, 2024
Maintainer Author

dennyglee Sep 6, 2024
Maintainer Author

dennyglee
Sep 6, 2024
Maintainer Author

dennyglee Sep 10, 2024
Maintainer Author

dennyglee Sep 10, 2024
Maintainer Author