Skip to content

Add new integration with Opik Tracking tool#11501

Merged
crazywoola merged 13 commits intolanggenius:mainfrom
comet-ml:opik-monitoring
Jan 13, 2025
Merged

Add new integration with Opik Tracking tool#11501
crazywoola merged 13 commits intolanggenius:mainfrom
comet-ml:opik-monitoring

Conversation

@Lothiraldan
Copy link
Contributor

@Lothiraldan Lothiraldan commented Dec 9, 2024

Summary

This PR integrates Opik (https://github.com/comet-ml/opik), an open-source platform for evaluating, testing, and monitoring LLM applications. It features built-in evaluation metrics and data annotation. Closes #11407, requested by a community user.

Screenshots

Before After
main branch

Checklist

Important

Please review the checklist below before submitting your pull request.

  • This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs/pulls)
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. 💪 enhancement New feature or request 📚 documentation Improvements or additions to documentation labels Dec 9, 2024
@Lothiraldan Lothiraldan mentioned this pull request Dec 9, 2024
Closed
@ZhouhaoJiang
Copy link
Contributor

ZhouhaoJiang commented Dec 12, 2024 •
edited
Loading

CleanShot 2024-12-12 at 15 43 28@2x
CleanShot 2024-12-12 at 15 53 23@2x
The elements have a position problem whenever I add another ops trace provider.

@ZhouhaoJiang
Copy link
Contributor

ZhouhaoJiang commented Dec 12, 2024 •
edited
Loading

CleanShot 2024-12-12 at 15 49 36@2x
CleanShot 2024-12-12 at 15 50 49@2x
When I add a credential, I don't fill in the Url field, because I see that it has a default value, but the actual request interface does not include this value, in order to better user experience should set it to the default value.

I will remove the api key after I finish test it.

@ZhouhaoJiang
Copy link
Contributor

ZhouhaoJiang commented Dec 12, 2024 •
edited
Loading

I have tested basic chat, agent chat, and workflow. There have no problems with data tracing.

@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Dec 12, 2024
@Lothiraldan
Copy link
Contributor Author

Lothiraldan commented Dec 12, 2024

@ZhouhaoJiang Thanks for your review. I fixed the issue for the default url, it was actually on the backend. I'm guessing that the frontend was sending an empty string for it.

I'm not sure to understand your feedback about the position of the Opik Tracing Provider. I added Opik at the end of both lists, the first one with configured tracing providers and the second one with non-configured tracing providers. Do you prefer a different behavior?

@ZhouhaoJiang
Copy link
Contributor

ZhouhaoJiang commented Dec 12, 2024

@ZhouhaoJiang Thanks for your review. I fixed the issue for the default url, it was actually on the backend. I'm guessing that the frontend was sending an empty string for it.

I'm not sure to understand your feedback about the position of the Opik Tracing Provider. I added Opik at the end of both lists, the first one with configured tracing providers and the second one with non-configured tracing providers. Do you prefer a different behavior?

I think my explanation is not clear. What I mean is that after one provider is configured, there is no gap between the components of the other two providers, and there is a problem with the ui

@Lothiraldan
Copy link
Contributor Author

Lothiraldan commented Dec 13, 2024

@ZhouhaoJiang I was able to fix the spacing issue but I realized that the Opik provider is missing the "IN USE" tag, I will check it out and fix the merge conflict on api/poetry.lock

Screenshot 2024-12-13 at 10-30-36 Long Story Generator (Iteration) - Dify
Screenshot 2024-12-13 at 10-30-19 Long Story Generator (Iteration) - Dify

@Lothiraldan
Copy link
Contributor Author

Lothiraldan commented Dec 13, 2024

I fixed the merged conflict on poetry.lock and finally understood that you can have only a single provider enabled per project. That's why I only saw only one provider with the "IN USE" tag. I think I fixed everything, let me know otherwise

ZhouhaoJiang
ZhouhaoJiang requested changes Dec 16, 2024
View reviewed changes
Copy link
Contributor

@ZhouhaoJiang ZhouhaoJiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have updated the workflow run id as workflow trace id and move the workflow app log id into metadata, So please update the workflow trace.

   def workflow_trace(self, trace_info: WorkflowTraceInfo):
        dify_trace_id = trace_info.message_id or trace_info.workflow_app_log_id or trace_info.workflow_run_id
        opik_trace_id = uuid4_to_uuid7(trace_info.start_time, dify_trace_id)

        if trace_info.message_id:
            trace_data = {
                "id": opik_trace_id,
                "name": TraceTaskName.MESSAGE_TRACE.value,
                "start_time": trace_info.start_time,
                "end_time": trace_info.end_time,
                "metadata": wrap_metadata(trace_info.metadata, message_id=trace_info.message_id),
                "input": wrap_dict("input", trace_info.workflow_run_inputs),
                "output": wrap_dict("output", trace_info.workflow_run_outputs),
                "tags": ["message", "workflow"],
                "project_name": self.project,
            }
            self.add_trace(trace_data)

You can refer to this pr #11585

@Lothiraldan
Copy link
Contributor Author

Lothiraldan commented Dec 16, 2024

@ZhouhaoJiang Thanks, I will update the workflow trace method accordingly. Can you remind me in which case the message_if field in the WorkflowTraceInfo could be None? I want to double-check that everything works nicely after the update.

iamjoel
iamjoel previously approved these changes Dec 17, 2024
View reviewed changes
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Dec 17, 2024
@Lothiraldan
Copy link
Contributor Author

Lothiraldan commented Dec 17, 2024

@ZhouhaoJiang I've updated the Opik tracer to use workflow_run_id or message_id when present.

ZhouhaoJiang
ZhouhaoJiang previously approved these changes Dec 18, 2024
View reviewed changes
@laipz8200
Copy link
Member

laipz8200 commented Dec 19, 2024

Thank you for this work, we'll test this feature in our development environment and may need to do some work on documentation.

@Lothiraldan
Copy link
Contributor Author

Lothiraldan commented Jan 6, 2025

@laojianzi Let me know if there is anything I can do on the documentation or the code. Do you want me to take care of the poetry.lock conflict?

@crazywoola
Copy link
Member

crazywoola commented Jan 8, 2025

@Lothiraldan Can you rebase the origin/main branch and push it again. I will merge it later.

@crazywoola crazywoola self-assigned this Jan 8, 2025
@Lothiraldan
Copy link
Contributor Author

Lothiraldan commented Jan 8, 2025

@crazywoola Sure, I rebased and pushed

crazywoola
crazywoola previously approved these changes Jan 8, 2025
View reviewed changes
@crazywoola
Copy link
Member

crazywoola commented Jan 8, 2025

Hello, it seems that the tests are failed.

Please add this to this file https://github.com/langgenius/dify/blob/d91960905971316ef27d7da5777cfa71ac85e521/web/i18n/zh-Hans/app.ts as well

 opik: {
      title: 'Opik',
      description: '一个全方位的开发者平台，适用于 LLM 驱动应用程序生命周期的每个步骤。',
 },

@Lothiraldan
Copy link
Contributor Author

Lothiraldan commented Jan 8, 2025

@crazywoola I see it, I'm working on resolving it now. Will also update this file.

@crazywoola crazywoola mentioned this pull request Jan 10, 2025
Closed
5 tasks
@crazywoola crazywoola merged commit 69d58fb into langgenius:main Jan 13, 2025
9 checks passed
@laipz8200 laipz8200 mentioned this pull request Jan 13, 2025
Merged
Scorpion1221 added a commit to yybht155/dify that referenced this pull request Jan 14, 2025
@Scorpion1221
Merge commit '6e0fb055d18969eb923e719ad92ecac3a5c5d534'
1597a07 
* commit '6e0fb055d18969eb923e719ad92ecac3a5c5d534': (42 commits)
  chore: bump version to 0.15.1 (langgenius#12690)
  feat: add table of contents to Knowledge API doc (langgenius#12688)
  [fix] support feature restore (langgenius#12563)
  api tool support multiple env url (langgenius#12249)
  Add new integration with Opik Tracking tool (langgenius#11501)
  fix: add type hints for App model and improve error handling in audio services (langgenius#12677)
  fix: Update variable handling in VariableAssignerNode and clean up app_dsl_service (langgenius#12672)
  Revert "Feat/new saas billing" (langgenius#12673)
  fix(workflow): fix answer node stream processing in conditional branches (langgenius#12510)
  fix: ruff with statements (langgenius#12578)
  fix: ruff check for True if ... else (langgenius#12576)
  chore: Adjust translations to align with Taiwanese Mandarin conventions (langgenius#12633)
  Fix pandas indexing method for knowledge base imports (langgenius#12637) (langgenius#12638)
  Feat/new saas billing (langgenius#12591)
  improve the readability of the function generate_api_key (langgenius#12552)
  chore: translate i18n files (langgenius#12543)
  Feat/add knowledge include all filter (langgenius#12537)
  fix: Add datasets list access control and fix datasets config display issue (langgenius#12533)
  fix: sum costs return error value on overview page (langgenius#12534)
  feat: show workflow running status (langgenius#12531)
  ...

# Conflicts:
#	api/poetry.lock
alexcodelf pushed a commit to alexcodelf/dify that referenced this pull request Jan 21, 2025
@Lothiraldan @alexcodelf
Add new integration with Opik Tracking tool (langgenius#11501)
3a533c2
@Gaocarri Gaocarri mentioned this pull request Feb 10, 2025
Closed
5 tasks
jsincorporated pushed a commit to jsincorporated/asaAi that referenced this pull request Jul 8, 2025
@Lothiraldan @jsincorporated
Add new integration with Opik Tracking tool (langgenius#11501)
9c2e9ca
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@crazywoola crazywoola crazywoola approved these changes

@iamjoel iamjoel iamjoel left review comments

@ZhouhaoJiang ZhouhaoJiang ZhouhaoJiang left review comments

@laipz8200 laipz8200 Awaiting requested review from laipz8200

Assignees

@crazywoola crazywoola

Labels

📚 documentation Improvements or additions to documentation 💪 enhancement New feature or request lgtm This PR has been approved by a maintainer size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add support for new Ops Tool, Opik

5 participants

@Lothiraldan @ZhouhaoJiang @laipz8200 @crazywoola @iamjoel