-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Description
Feast 0.10 does not support ingestion to the online store from streams like Kafka. This functionality was available in 0.9, and it's on the roadmap for Feast to support this with the FeatureView class before the 0.11 release.
Our proposed API is just to maintain the Feast existing Feast 0.9 API for defining ingestion from streams. This looks like:
driver_ft = FeatureView(
name="driver_trips",
entities=["driver_id"],
features=[
Feature("average_daily_rides", ValueType.FLOAT),
Feature("rating", ValueType.FLOAT)
],
max_age=Duration(seconds=3600),
batch_source=BigQuerySource(
table_ref="gcp_project:bq_dataset.bq_table",
event_timestamp_column="datetime",
created_timestamp_column="timestamp",
field_mapping={
"rating": "driver_rating"
}
),
stream_source=KafkaSource(
...
)
)
We also considered a decorator-based API:
kafka_source = KafkaSource(...)
driver_ft = FeatureTable(
name="driver_trips",
entities=["driver_id"],
features=[
Feature("average_daily_rides", ValueType.FLOAT),
Feature("rating", ValueType.FLOAT)
],
max_age=Duration(seconds=3600),
inputs={"orders": Input(source=kafka_source, window="7d")},
)
The main benefit of this decorator-based API is that it allows for the expression of more complex logic. We are designing the API for OnDemandFeatures in parallel (this discussion on OnDemandFeatures will happen on a public issue to be created soon.) A decorator-based approach could be nice for OnDemandFeatures.
However, this decorator approach doesn't look like it adds value to streaming feature declarations and might be overkill, so the proposal is to stick with the Feast 0.9 API.