Can we add some documentation under /genai/prompt-registry/ as well? Maybe it's worth having a single page explains link between prompt and traces. Maybe follow-up after adding UI feature to jump to a list of traces from the prompt page.

Yeah, let me add the doc section after adding UI changes.

Can we actually use the entity association somehow? Since traces are stored in the experiment, search query goes through tracking store anyway.

We could migrate the linkage to EntityAssociation, but what the motivation would be? For storing more structured linkage?

For performance. Entity association table stores one row per prompt+trace combination, ooposed to a tag that combines multiple prompts into one row, so we can use the exact match condition and benefit from the index of destination ID (WHERE destination_type = 'prompt' and destination_id = '<name/version>'). This should be much more efficient than LIKE on tag values.

Understood, do we think the efficiency gain is worth breaking the backward compatibility right now? Current search does not require any data model change, and we can always switch to entity association if users report search slowness.

Understood, do we think the efficiency gain is worth breaking the backward compatibility right now?

Yes. Afaik we don't provide any place users can see the link between tag and prompt in OSS side, then I don't think it breaks anything. The migration becomes harder when we do it after adding the search functionality.

If you mean Databricks side, we can keep tag logging for DBX. They also plan to migrate to entity association table fwiw.

I think consolidating the linkage in entity association table and populate trace fields (either tag or metadata) from the entity association is clearer since we can keep single source of truth.

Yes, it removes duplication, but one downside is every prompt load needs to make additional SQL query, even if the trace is not linked to any prompt. Since tags are stored in TraceInfo, searching traces also triggers that for every prompt.

Also unfortunately Databricks UI already shows prompts link based on trace tag (which is different from OSS). This means that, even after dbx migrate to entity table, we cannot remove the tag dependence at least for a certain period😔

This means that, the end state will be

Option 1 (yours):

• Write: Set the tag in dbx only. Store a record in entity association table.
• Read: Populate tag/metadata at load time.

Option 2:

• Write: Set the tag and store to entity association table always.
• Read: none

I'm not sure if the state 1 is cleaner than 2. The single source of truth is nice in general, but not always the best - for example, we can think it as a kind of databse denormalization - we denormalize the info to entity table for search optimization.

I got your point, but do you think the single additional query for retrieving associations is that problematic? As you said earlier, it should be quick since it relies on database indices, and as I implemented, it does not cause N+1 issue by using batch loading. Btw not necessarily objecting to option 2, we can go with dual writing if skipping one query is really worth database denormalization.

Btw, there are several todo comments to remove tag based linkage, didn't we want to remove the tag based approach?

I'm fine with populating it to tags if you feel strongly about it. For the TODO comment, I didn't expect to have "linked prompt" tab in the trace UI but just to have a button like "see linked prompts" that jumps to search page:)

I see, shall we defer the decision by implementing option 2 for now? Option 2 -> option 1 is feasible while the opposite is not.