Skip to content

Ume datamodule - allow downloads of HF datasets#50

Merged
ncfrey merged 12 commits intomainfrom
ume-runs
Mar 21, 2025
Merged

Ume datamodule - allow downloads of HF datasets#50
ncfrey merged 12 commits intomainfrom
ume-runs

Conversation

@karinazad
Copy link
Collaborator

No description provided.


paths:
root_dir: ./runs
root_dir: /data/lobster/ume
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update to use env variable

shuffle_buffer_size: 1000
num_workers: 4
seed: 0
sample: false # if false, uses RoundRobinConcatIterableDataset, else MultiplexedSamplingDataset
Copy link
Collaborator Author

@karinazad karinazad Mar 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed comments since docstrings are in the Ume datamodule

@ncfrey ncfrey merged commit a69b771 into main Mar 21, 2025
5 checks passed
@ncfrey ncfrey deleted the ume-runs branch March 21, 2025 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants