Hello @woop @YikSanChan,

FYI, I have implemented part of the Hive offline store, just want have a catch up with you, and probably get some feedbacks before continue.
could you have a look when you got time ?

The repo is here: https://github.com/baineng/feast-hive

Basically it's very similar to BigQuery implementation.

Some thoughts I have when checking Feast code:

The DataSource abstraction in source code are separated from offline_store and online_store, but they are actually coupled (BigQuery store relies on BigQuerySource, ...), probably DataSources should stay together with their implementations.

And some code about DataSource are hardcoded, not easy to extend. Such as DataSource.from_proto(data_source).

Some questions:

Should I add protos for HiveSource and HiveOptions in my repo as well?

I saw you guys have some discussion about removing query in the BigQuerySource, what do you think if i remove it from HiveOfflineStore ? (Hive should have more controls by end-users, I mean creating views)

I saw in get_historical_features() of BigQueryOfflineStore, it uploads entity_df to BigQuery first, then do point-in-time query.
Think I will need to do the same for Hive, but there is no efficient way to upload a df to Hive besides uploading to HDFS directly. I don't want have HDFS client as another dependency, so I will choose to use multiple rows insert. what do you guys think?

This is so awesome @Baineng. Thank you for working on Hive support!

The DataSource abstraction in source code are separated from offline_store and online_store, but they are actually coupled (BigQuery store relies on BigQuerySource, ...), probably DataSources should stay together with their implementations.

Yes, we agree. Needs to be cleaned up.

And some code about DataSource are hardcoded, not easy to extend. Such as DataSource.from_proto(data_source).

Yea, we realize this as well. Good catch. It needs to be generalized.

Should I add protos for HiveSource and HiveOptions in my repo as well?

I believe the answer is yes. We need to store the source/config in the registry so it needs to exist, and I dont think it should be in the main repo.

I saw you guys have some discussion about removing query in the BigQuerySource, what do you think if i remove it from HiveOfflineStore ? (Hive should have more controls by end-users, I mean creating views)

I prefer not to introduce query.

saw in get_historical_features() of BigQueryOfflineStore, it uploads entity_df to BigQuery first, then do point-in-time query.
Think I will need to do the same for Hive, but there is no efficient way to upload a df to Hive besides uploading to HDFS directly. I don't want have HDFS client as another dependency, so I will choose to use multiple rows insert. what do you guys think?

You do have to do the same, but I share your intuition. It would be nice if we didn't couple to HDFS but instead wrote directly to Hive. Also, we can allow users to provide a reference to a table where their entity df is available within Hive, so we can skip the upload step. So at least there is a way for them to use Hive.

Add Hive Support #1686

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions