Skip to main content

The basics

Shaped is a real-time retrieval engine that lets you build state-of-the-art retrieval systems. You can:

  • Import from 20+ connectors
  • Transform data with SQL or AI into materialized views for training
  • Define retrieval engines, models, and ranking pipelines as YAML files
  • Test different models, ranking strategies, and configurations in parallel
  • Index on new data automatically and in real-time
  • Scale up or down automatically based on usage

Architecture overview​

Shaped's architecture is grouped into three layers: Data, Intelligence, and Training.

Each layer serves a specific purpose and builds upon the previous one:

  1. Data layer (Tables and Views) forms the foundation, connecting to your existing data sources via connectors. Tables store raw data with defined schemas, while Views transform and enrich this data through SQL queries or AI-powered enrichment. This layer serves as the data warehouse that feeds into the intelligence layer.

  2. Intelligence layer (Engines) consumes data from tables and views to build retrieval systems. Engines define how data is indexed (through embeddings for vector search, lexical search for BM25, or by configuring specific column indices for user and item tables, such as btree, bitmap, label_list, or fts), which scoring models are trained (from simple popularity-based models to complex machine learning models), and how the system is deployed and scaled. This layer handles all the model training, embedding generation, and indexing logic.

  3. Query layer executes retrieval operations against your engines through a REST API. Each query follows a four-stage pipeline: retrieve candidates, filter based on business rules, score using trained models, and reorder for diversity. This layer provides the real-time interface for retrieving and ranking results.

Data flows through Shaped in a pipeline: your production data sources connect to Tables via connectors. Views transform this raw data into the shape needed for your use case. Engines then ingest data from tables and views, train models, and build indexes. Finally, Queries execute against engines to retrieve and rank results in real-time.

Data layer​

This is the data warehouse. You can connect production datasets, transform and clean the data into a usable format, join disconnected sources to create unified datasets, and enrich your datasets with LLMs.

Import data with a connector​

You can import data from an external source using one of our 20+ connectors. If you need a custom table, you can declare a custom table and add rows to it via API.

Tables keep data synced​

Once your data is imported, it exists in Shaped as tables. Depending on the connector you use, new data is updated in real-time (streams) or every 15 minutes (batch).

DATA SOURCESCONNECTORSTABLESClickHouseOLAP databaseSnowflakeData warehouseAmazon S3Blob storageAmplitudeEvent analyticsCONNECTOR LAYERReal-timeStreamingBatch15 min refreshAPI insertDirect writes20+ pre-built connectorsPostgres | Kafka | S3 | BigQuerySegment | Redshift | MongoDB | ...Raw synced dataproductsid | sku | price | categoryproduct_imagesproduct_id | url | altuser_clicksuser_id | item_id | tsuser_preferencesuser_id | likes | tagsagent_conversationssession_id | turn | messageuser_chat_memoriesuser_id | memory | ts

Views: clean, enrich, and join your data​

Use views to unify disparate data sources. If your tables are not in the right structure to be trained on, you can use views to fix or enrich them into new views.

TABLESVIEWSENGINESRaw synced dataCleaned | joined | enrichedRetrieval layerproductsid | sku | price | categoryproduct_imagesproduct_id | url | altuser_clicksuser_id | item_id | tsuser_preferencesuser_id | likes | tagsagent_conversationssession_id | turn | messageuser_chat_memoriesuser_id | memory | tsimage_descriptionsAI-generated captionsFINALenriched_productsJoined product metadatawith image descriptionsFINALuser_events_l30dRolling 30-day joinFINALchat_contextUnified table of memory and chatTRAINING +INFERENCEVector encodingAgent retrievalModel trainingRankingLow-latencyReal-time serving

There are two types of views: SQL views, and AI enrichment views.

SQL views​

SQL views let you:

  • Convert data types, fix null values, or rename columns
  • Join multiple tables into one massive interactions table.
  • Combine multiple sources into one table - eg user events from your product with billing transactions)

AI enrichment views​

AI enrichment views let you:

  • Add new semantic data from existing columns
  • Create new columns like clothing_color from an image of clothes

Each view is materialized, so you can use the outputs to train a model (or transform in a different way).

Intelligence layer​

The intelligence layer is where indexing, embedding generation, and model training happens. You will declare an engine which contains the scoring and ranking logic for your retrieval system.

Engines​

When you configure an engine, you can define multiple retrieval components that will be trained together. You will also describe what data the engine is connected to and deployment configuration (such as number of pods, data tier, etc).

Your engine does not have to be limited to a single embedding or scoring model. You can configure multiple models, plus embeddings for semantic search, plus a custom lexical/bm25 search engine, all in one engine. Each component has an API to fine-tune it accordingly.

You can also run multiple engines in parallel to test the best one.

Query layer​

The query layer is how you retrieve data using your engine. It is a REST API with a simple configuration to handle multiple different retrieval strategies with varying complexity.

At the simple end, you can do standard BM25 lexical search with a text query.

At the more complex end, you can run complex 4-stage ranking pipelines with retrieval, filtering, scoring, and ordering steps.

You can write queries using ShapedQL, a SQL-like domain-specific language that provides familiar SQL syntax for building recommendation and ranking queries. ShapedQL supports all retrieval strategies, from simple similarity searches to complex multi-stage ranking pipelines with custom scoring expressions.

tip

If you want to learn more about the four-stage recommender model, check out our series: Anatomy of Modern Ranking Architectures