Highlights

⚡ The new v0.7.0 release introduces major efficiency improvements to the fev package.

Several operations such as past/future splits and metric computations now have efficient in-memory implementations based on pyarrow, instead of slow parallel processing based on datasets.Dataset.map. This significantly reduces the end-to-end runtime of the fev-bench benchmark, and avoid unnecessary caching in the ~/.cache/huggingface/datasets directory.

What's Changed

• Update contrib docs by @shchur in #51
• Add task definitions and results for fev_bench by @shchur in #52
• Update task definitions by @shchur in #54
• Fix tutorial and leaderboard links in README by @abdulfatir in #56
• Split CSV files per model by @shchur in #57
• Add Toto and TimesFM-2.5 examples by @abdulfatir in #53
• Add Chronos-2 example by @abdulfatir in #58
• Add badges and bibtex to readme by @abdulfatir in #59
• Revise leaderboard details in README.md by @abdulfatir in #61
• Make dependabot ignore examples/ folder by @shchur in #62
• Fix dataset_path in task summaries by @shchur in #74
• Normalize runtime by number of forecasts by @shchur in #75
• Enable CI with GitHub actions by @shchur in #78
• Fix time normalization issues by @shchur in #79
• Disable optional updates from dependabot by @shchur in #81
• Remove dependabot by @shchur in #82
• Speed up past/future split by @shchur in #83
• Speed up seasonal error calculation by @shchur in #85
• Do not cache splits as attributes of the EvaluationWindow by @shchur in #86
• Fix seasonal scaling in MASE/SQL/RMSSE by @shchur in #89

Full Changelog: v0.6.1...v0.7.0

What's Changed

• Rerun notebooks with v0.6.0 by @shchur in #47
• Add leakage_imputation_model by @shchur in #48

Full Changelog: v0.6.0...v0.6.1

🚨 Major Breaking Changes

The new 0.6.0 release contains lots of improvements and breaking changes to the API. For existing users, we recommend taking a look at the updated tutorials to get familiar with the changes and the new functionality.

• Task API Redesign: Task now contains multiple rolling windows, TaskGenerator class is deprecated.

  New code for evaluation on multiple rolling windows:

  task = fev.Task(..., num_windows=2)
  predictions_per_window = []
  for window in task.iter_windows():
      past_data, future_data = window.get_input_data()
      predictions_per_window.append(model.predict(past_data, future_data))
  
  # A single summary with average score across evaluation windows
  summary = task.evaluation_summary(predictions_per_window, model_name="my_model")

  Old code (version <=0.5.0)

  task_generator = fev.TaskGenerator(..., num_rolling_windows=2)
  for task in task_generator.generate_tasks():
      past_data, future_data = task.get_input_data()
      predictions = model.predict(past_data, future_data)
      # One summary per evaluation window
      summary = task.evaluation_summary(predictions, model_name="my_model")

• Renamed parameters: Deprecated Task/TaskGenerator parameter names:

  • num_rolling_windows → num_windows
  • rolling_step_size → window_step_size
  • cutoff → initial_cutoff
  • target_column → target
  • multiple_target_columns → generate_univariate_targets_from
  • lead_time → ignored
  • excluded_column_names → ignored

• Covariate specification: Names of covariate columns now need to provided explicitly during Task creation using known_dynamic_columns, past_dynamic_columns, static_columns keyword arguments. Columns present in the dataset

• Leaderboard changes: fev.leaderboard() now uses skill_score and win_rate instead of gmean_rel_error and avg_rank. The method also optionally returns 95% confidence intervals based on bootstrap.

• Updated results and benchmark definitions on GitHub: Benchmark task definitions, results, and model wrappers on the main branch have been updated to be compatible with v0.6.0. Please view the old branches (e.g., v0.5.0) to access the old task definitions, results and model wrappers.

What's Changed

• Add multivariate <-> univariate conversion support by @shchur in #27
• Refactor Task class to include multiple rolling window evaluations by @shchur in #33
• Make metrics customizable by @canerturkmen in #35
• Update model wrappers and results to be compatible with the new Task design
• Update README.md by @shchur in #26
• Speed up fev.utils.validate_time_series_dataset by @shchur in #29
• Implement DartsAdapter by @shchur in #30
• Use ordinal encoding for dynamic categorical features in GluonTSAdapter by @shchur in #31
• Make covariate column names explicit by @shchur in #34
• Minor fixes to Task by @shchur in #37
• Fix Task.to_dict() with dict-based metrics by @shchur in #38
• Refactor analysis code by @shchur in #36
• Add results for TimeCopilot by @AzulGarza in #39
• Add static webpages with documentation using mkdocs by @shchur in #42
• Update results to v0.6.0 by @shchur in #43
• Update model wrappers for v0.6.0 by @shchur in #44
• Update readme for v0.6.0 by @shchur in #45

New Contributors

• @canerturkmen made their first contribution in #35
• @AzulGarza made their first contribution in #39

Full Changelog: v0.5.0...v0.6.0

This is a pre-release version.

What's Changed

• Refactor analysis code by @shchur in #36
• Add static webpages with documentation using mkdocs by @shchur in #42
• Update results to v0.6.0 by @shchur in #43
• Update model wrappers for v0.6.0 by @shchur in #44
• Update readme for v0.6.0 by @shchur in #45

Full Changelog: v0.6.0rc3...v0.6.0rc4

This is a pre-release version.

What's Changed

• Use ordinal encoding for dynamic categorical features in GluonTSAdapter by @shchur in #31
• Refactor Task class to include multiple rolling window evaluations by @shchur in #33
• Make covariate column names explicit by @shchur in #34
• make metrics customizable by @canerturkmen in #35
• Minor fixes to Task by @shchur in #37
• Fix Task.to_dict() with dict-based metrics by @shchur in #38

New Contributors

• @canerturkmen made their first contribution in #35

Full Changelog: v0.6.0rc2...v0.6.0rc3

This is a pre-release version.

What's Changed

• Minor fixes and optimizations by @shchur in #29
• Implement DartsAdapter by @shchur in #30

Full Changelog: v0.6.0rc1...v0.6.0rc2

This is a pre-release version.

What's Changed

• Update README.md by @shchur in #26
• Add multivariate <-> univariate conversion support by @shchur in #27

Full Changelog: v0.5.0...v0.6.0rc1

Highlights

• Multivariate forecasting support: The users can now create multivariate tasks by setting the target_column to a list[str] with names of target columns when creating a Task. Check out the updated tutorial for more details.
• API changes: The following Task attributes have been deprecated. Results containing old names can still be used, but creating new tasks with deprecated attributes will produce a warning.
  • multiple_target_columns has been renamed to generate_univariate_targets_from.
  • min_ts_length has been replaced by min_context_length.
• Improved handling of short series: Previously, if some time series in the dataset were too short for the chosen horizon and cutoff combination, and exception would be raised. Now, these series will be automatically filtered out during dataset loading.

Changelog

• Add support for multivariate forecasting in Task by @shchur in #15
• Improve time series filtering based on cutoff, horizon and min_context_length by @shchur in #18
• Add TiRex results by @apointa in #17
• Fix seasonal differences for short series by @shchur in #20
• Expose prediction validation as a public method by @abdulfatir in #21
• Handle DatasetDict in clean_and_validate_predictions by @abdulfatir in #22

New Contributors

• @apointa made their first contribution in #17
• @abdulfatir made their first contribution in #21

Full Changelog: v0.4.1...v0.5.0

This is a pre-release version.

Changes:

• Expose prediction validation as a public method (#21)
• Handle DatasetDict in clean_and_validate_predictions (#22)

This is a pre-release version.

Changes:

• Fix seasonal differences for short series (#20)