Skip to content

Conversation

@tamarl08
Copy link
Contributor

@tamarl08 tamarl08 commented Sep 6, 2023

  • log all the values of the following tables as run metrics: consistency, eval_short, true_score_eval. This allows easy comparison between runs in the runs view

  • log confusion matrices as custom charts, this also allows comparing between runs - see in the workspace view

@desilinguist @mulhod The changes make the tutorial run quite slow for me. Please test and let me know what you think.
to test add the following to rsmtool / rsmeval tutorial config (change project name accordingly), and run it in this branch:

    "use_wandb": true,
    "wandb_project": "rsmtool_tutorial",
    "wandb_entity": "etslabs"

@codecov
Copy link

codecov bot commented Sep 7, 2023

Codecov Report

Patch coverage is 100.00% of modified lines.

Files Changed Coverage
rsmtool/rsmeval.py ø
rsmtool/rsmtool.py ø
rsmtool/analyzer.py 100.00%
rsmtool/utils/wandb.py 100.00%

📢 Thoughts on this report? Let us know!.

@tamarl08 tamarl08 force-pushed the enhance_wandb_rsmeval branch from ffaa0b5 to d0e14f1 Compare September 7, 2023 14:26
@mulhod
Copy link
Contributor

mulhod commented Sep 7, 2023

LGTM!

Copy link
Collaborator

@desilinguist desilinguist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@tamarl08 tamarl08 merged commit 0684a3b into main Sep 7, 2023
@delete-merged-branch delete-merged-branch bot deleted the enhance_wandb_rsmeval branch September 7, 2023 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants