Problem:

Users should be able to serve pytorch models that were produced with Transformers4Rec or any other process using a Systems ensemble. This will work towards supporting Session-based models as well as expanding System's support for a new modeling framework.

Goal:

Systems should be able to serve all pytorch models that are currently supported by Triton.

Definition of Done

Have an example that serves PyT session based model in conjunction with a NVT workflow where the session based models scores the whole catalog

Open questions

Constraints:

Not all pytorch models can be served via Triton's pytorch backend, so we will need to be able to use multiple backends in order to serve all Triton-compatible pytorch models.

Starting Point:

Transformers4Rec

• Update T4R to use Merlin std schema
  • Add save/load & input/output schema methods to T4Rec Model class  Transformers4Rec#507
• [Task] Transformers4Rec Serving using Merlin Schema Transformers4Rec#499
• Have a padding operator and add Support for ragged inputs that we can use consistently with Transformers for rec systems#322
• [Task] Add topK layer in model to return topK items at inference step Transformers4Rec#668

Systems

• Use Triton's pytorch backend for "torchscriptable" models for optimized performance
  • Create a PredictPytorch operator systems#153
  • Write an integration test for T4R Torchscript serving systems#176
• Load the schema files written out by T4R

Integration Issues

• Address the issues @rnyak surfaced while trying to put together an example
  • Fix missing value_count property and/or error from value_count op if sequence min and max length is same: This PR Remove specifying is_ragged in LocalExecutor _transform_data core#173 should fix that issue.
  • Fix wrong response dimension from TIS after sending request. This PR Refactoring part1- flags modification Transformers4Rec#543 should be fixing that issue.
  • Pad the rows when converting dataframes to dictionaries for Triton requests systems#234
  • [FEA] Create an easy functionality to generate dict of tensors- a standard way to move array data across frameworks core#175
  • [BUG] getting error when serving Session-based NVT workflow model on TIS with TransformWorkflow op  systems#240

Nice to have: (P1)

• [FEA] Add list column support in Merlin Systems systems#135

Documentation

• [Task] Add Example for Transformer4Rec Serving Example Transformers4Rec#491
• [BUG] Bugs to deploy NVTabular + Transformer4Rec with TorchScript backend systems#257
• Review API documentation for PyT serving support for T4R models #904
• Add/Fix the doc strings for PyT serving support for T4R models in TorchScript Transformers4Rec#692
• [DOC] Improve Code Coverage in Transformers4Rec to X% Transformers4Rec#664

Examples

• Deprecate the old example and update example for PyT serving support for T4R models with systems support and Torch script backend Transformers4Rec#660
• [Task] Update Legacy Inference Example to use Merlin Systems + TorchScript Transformers4Rec#676

Blockers:

• [INF] Unresolved architectural decisions
• Support for ragged tensors in T4R
• Start with fix padding and have it all padded to the same length and then investigate whether it's worthwhile to add the like padding to the Max Sequence length
• Padding support in dataloader that supports systems along with cross framework support