### Tasks
- [ ] Add a draft title or issue reference here

Problem:

Customers need a way to load embeddings that have been pretrained or trained from separate models into the model.
See #471

Goal:

Enable dataloading of separate embedding tables without having to add these embeddings into the interaction data during training. For serving those embeddings need to be provided in the request to the model. The feature must be ueseable in production setting

Constraints:

• External embedding tables may not fit on GPU.
• Non-trainable embeddings
• Fits in CPU memory, Larger than CPU memory is left for potential future work
• Not generating the embedding on the fly (future work)

Supporting pre-trained vector embeddings as features would provide baseline support for multi-modal use cases that rely on outside models to generate image/text embeddings.

NVTabular

• Refactor Categorify NVTabular#1692
• Update the c++ versions of categorify serving to match the new functionality
• [BUG] Simplify Categorify encoding for better standardization and easier reverse mapping NVTabular#1748
• Enable re-use of a fitted Categorify operator in another workflow #972
• NVTabular saving index+embedding tables as a numpy array file #971
• Feed pre-trained embeddings to NVTabular
  Is this part of this RMP ticket?

Core

• Add a tag for pre-trained embeddings core#238

Dataloader

• Add lookup for embeddings based on key during dataloading dataloader#31
• Add pretrained embedding to the dictionary of tensors dataloader#32
• Support for 3d tensors (fixed size of sequence) to support session based in TF4Rec and MM dataloader#34
• Modify the padding operator to only allow padding values of 0 (in conjunction with the changes to categorify)

Transformers4Rec

These features under T4R will not be in scope for this RMP ticket. The development will happen in Models.
PR implementing pre-trained support in T4Rec: NVIDIA-Merlin/Transformers4Rec#690

• Pre-trained embeddings support: Set the models schema from the dataloader output schema Transformers4Rec#682
• Update TabularFeatures to support input from pretrained embeddings, with optional projection Transformers4Rec#683
• Update TabularSequenceFeatures to support input from pretrained embeddings, with optional projection Transformers4Rec#684
• Convert pre-trained embeddings into 2D/3D dense tensor representation Transformers4Rec#685
• Evaluate the transforms that happen to an embedding after it's pulled from the embedding table so that pretrained embeddings can be processed in the same way. Transformers4Rec#500
• [Task] Support pre-trained vector embeddings as input features  Transformers4Rec#485

Related PR: NVIDIA-Merlin/Transformers4Rec#690

Models (TF API)

PR #1083 implementing pre-trained support in MM

• Convert sequential pre-trained embeddings into 3d ragged tensor representation and ensure SequenceTransform accepts 3D ragged tensor models#1071
• Create a PretrainedEmbeddings block and integrate it with InputBlockV2. models#1068
• Ensure that BroadcastFeatures support expanding 2D pre-trained embeddings (contextual features) to concat with 3D sequence features models#1073
• Adapt SequenceMasking to support pre-trained embeddings models#1070
• Update models ranking models (DLRM, DCN) to support PretrainedEmbeddings with fixed dim projection. models#1072

Merlin Systems

• Support pre-trained vector embeddings as features - Add embedding to the recommendation request (from feature store, KV, etc?) systems#210

Examples

• Example of dataloading pretrained embeddings in Transformers4Rec Transformers4Rec#501
• Example of dataloading pretrained embeddings in Merlin Models models#788
• Example for serving pre-trained vector embeddings as features #886
• How to for embedding op with categorify from scratch NVTabular#1827 ( This ticket closes all NVT work )

Documentation

• Document the API of pretrained embeddings #1000
• Document the limitations of our pretrained solution #1001