Skip to content

StreamingFeatureView API for 0.10  #1564

@jparthasarthy

Description

@jparthasarthy

Feast 0.10 does not support ingestion to the online store from streams like Kafka. This functionality was available in 0.9, and it's on the roadmap for Feast to support this with the FeatureView class before the 0.11 release.

Our proposed API is just to maintain the Feast existing Feast 0.9 API for defining ingestion from streams. This looks like:

driver_ft = FeatureView(
    name="driver_trips",
    entities=["driver_id"],
    features=[
      Feature("average_daily_rides", ValueType.FLOAT),
      Feature("rating", ValueType.FLOAT)
    ],
    max_age=Duration(seconds=3600),
    batch_source=BigQuerySource(
        table_ref="gcp_project:bq_dataset.bq_table",
        event_timestamp_column="datetime",
        created_timestamp_column="timestamp",
        field_mapping={
          "rating": "driver_rating"
        }
    ),
    stream_source=KafkaSource(
	  ...
    )
)

We also considered a decorator-based API:

kafka_source = KafkaSource(...) 

driver_ft = FeatureTable(
    name="driver_trips",
    entities=["driver_id"],
    features=[
      Feature("average_daily_rides", ValueType.FLOAT),
      Feature("rating", ValueType.FLOAT)
    ],
    max_age=Duration(seconds=3600),
		inputs={"orders": Input(source=kafka_source, window="7d")},
)

The main benefit of this decorator-based API is that it allows for the expression of more complex logic. We are designing the API for OnDemandFeatures in parallel (this discussion on OnDemandFeatures will happen on a public issue to be created soon.) A decorator-based approach could be nice for OnDemandFeatures.

However, this decorator approach doesn't look like it adds value to streaming feature declarations and might be overkill, so the proposal is to stick with the Feast 0.9 API.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions