Skip to content

[ENH] Adding support for GluonTS' PandasDataset object#6668

Merged
fkiraly merged 21 commits intosktime:mainfrom
shlok191:gluonTS_pandasDataset
Jul 17, 2024
Merged

[ENH] Adding support for GluonTS' PandasDataset object#6668
fkiraly merged 21 commits intosktime:mainfrom
shlok191:gluonTS_pandasDataset

Conversation

@shlok191
Copy link
Copy Markdown
Contributor

@shlok191 shlok191 commented Jun 26, 2024

Reference Issues/PRs

Fixes #6658

What does this implement/fix? Explain your changes.

This implementation adds conversion functions to easily deal with pd.DataFrame of the long and long varieties in addition to lists and dictionaries of the same to gluonTS PandasDataset objects. This also includes the conversion of pd.Series to gluonTS PandasDataset objects.

Does your contribution introduce a new dependency? If yes, which one?

Nope! Since GluonTS is a pre-existing dependency added in from a different PR.

What should a reviewer concentrate their feedback on?

A reviewer should concentrate on the sktime/datasets/_adapter/gluonts.py file.

Did you add any tests for the change?

Not yet, but I am planning on doing so

Any other comments?

I was really hoping to add the gluonTS PandasDataset as a proper mtype, but I could not find any way to convert a PandasDataset to a pd.DataFrame. I'll discuss this more in a comment!

PR checklist

For all contributions
  • I've added myself to the list of contributors with any new badges I've earned :-)
    How to: add yourself to the all-contributors file in the sktime root directory (not the CONTRIBUTORS.md). Common badges: code - fixing a bug, or adding code logic. doc - writing or improving documentation or docstrings. bug - reporting or diagnosing a bug (get this plus code if you also fixed the bug in the PR).maintenance - CI, test framework, release.
    See here for full badge reference
  • The PR title starts with either [ENH], [MNT], [DOC], or [BUG]. [BUG] - bugfix, [MNT] - CI, test framework, [ENH] - adding or improving code, [DOC] - writing or improving documentation or docstrings.

@shlok191
Copy link
Copy Markdown
Contributor Author

shlok191 commented Jun 26, 2024

@fkiraly, I have added conversion functions to convert pandas DataFrames to the GluonTS PandasDataset that is utilized by LagLlama! I was hoping to make the new data format an sktime mtype, but unfortunately there is no way that I can find which converts a PandasDataset back to pd.DataFrame.

Could we have it possibly reference the use of the adapter functions we have defined in the LagLlama documentation and the written examples? That way, I think we could make the conversion functions known easily to end users. If there is a better way to proceed, do let me know and I'll update the PR!

@shlok191
Copy link
Copy Markdown
Contributor Author

shlok191 commented Jul 3, 2024

@fkiraly, Hello Franz! I have added support for the PandasDataset type along with pd-multiindex. I hope that the PR is okay, and I'm moving on to the LagLlama work now! :)

@fkiraly
Copy link
Copy Markdown
Collaborator

fkiraly commented Jul 3, 2024

@shlok191, after merging #6530, there are a lot of conflicts with this PR, can you kindly resolve?

@shlok191
Copy link
Copy Markdown
Contributor Author

shlok191 commented Jul 4, 2024

@fkiraly, I think the conflicts have been fixed. Thank you for pointing them out!

@shlok191 shlok191 mentioned this pull request Jul 12, 2024
Open
6 tasks
@fkiraly fkiraly added module:datatypes datatypes module: data containers, checkers & converters enhancement Adding new functionality labels Jul 17, 2024
fkiraly
fkiraly approved these changes Jul 17, 2024
View reviewed changes
Copy link
Copy Markdown
Collaborator

@fkiraly fkiraly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks!

@fkiraly fkiraly merged commit d3e568e into sktime:main Jul 17, 2024
@shlok191
Copy link
Copy Markdown
Contributor Author

shlok191 commented Jul 18, 2024

@fkiraly, no problem! Thanks a lot for merging it in! :)

yarnabrina added a commit to Abhay-Lejith/sktime that referenced this pull request Jul 20, 2024
@yarnabrina
Merge remote-tracking branch 'upstream/main' into pr/Abhay-Lejith/6732
23b4279 
* upstream/main:
  [ENH] Allow object dtype in series (sktime#5886)
  [ENH] `check_pdmultiindex_panel` to return names of invalid `object` columns if there are any (sktime#6797)
  [ENH] added feature_kind metadata in datatype checks (sktime#6490)
  [ENH] Adding tag for categorical support in `X` (sktime#6704)
  [BUG] Fix bug when predicting segments from clasp change point annotator (sktime#6756)
  [ENH] Adding support for GluonTS' PandasDataset object (sktime#6668)
Abhay-Lejith added a commit to Abhay-Lejith/sktime that referenced this pull request Jul 31, 2024
@Abhay-Lejith
Squashed commit of PR#6732 adding the following:
aa698b0 
commit 9f95d89
Merge: 02bd3fb 6caeb0d
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Tue Jul 30 22:01:03 2024 +0530

    Merge branch 'main' into step3-4_categorical_support

commit 02bd3fb
Merge: 53ee977 a21d97f
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Mon Jul 29 21:47:41 2024 +0530

    Merge branch 'main' into step3-4_categorical_support

commit 53ee977
Merge: 74fd3f8 9b066f2
Author: Franz Király <f.kiraly@ucl.ac.uk>
Date:   Sun Jul 28 15:19:13 2024 +0100

    Merge branch 'main' into pr/6732

commit 74fd3f8
Merge: e9f5e2b 116faa2
Author: Franz Király <f.kiraly@ucl.ac.uk>
Date:   Sun Jul 28 15:18:26 2024 +0100

    Merge branch 'main' into pr/6732

commit e9f5e2b
Author: Abhay-Lejith <Abhay-Lejith@users.noreply.github.com>
Date:   Sun Jul 28 10:32:15 2024 +0000

    [AUTOMATED] update CONTRIBUTORS.md

commit f3e212b
Merge: 95c213b 81d2d08
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sun Jul 28 16:01:12 2024 +0530

    Merge branch 'main' into step3-4_categorical_support

commit 95c213b
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sun Jul 28 15:59:02 2024 +0530

    added est.update to test_categorical

commit 8d14bde
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sat Jul 27 09:35:51 2024 +0530

    typo, docs

commit 81b4862
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Thu Jul 25 22:27:00 2024 +0530

    separate test for categorical in y

commit 23b4279
Merge: a0d950d 3a55ca5
Author: Anirban Ray <39331844+yarnabrina@users.noreply.github.com>
Date:   Sat Jul 20 09:29:27 2024 +0530

    Merge remote-tracking branch 'upstream/main' into pr/Abhay-Lejith/6732

    * upstream/main:
      [ENH] Allow object dtype in series (sktime#5886)
      [ENH] `check_pdmultiindex_panel` to return names of invalid `object` columns if there are any (sktime#6797)
      [ENH] added feature_kind metadata in datatype checks (sktime#6490)
      [ENH] Adding tag for categorical support in `X` (sktime#6704)
      [BUG] Fix bug when predicting segments from clasp change point annotator (sktime#6756)
      [ENH] Adding support for GluonTS' PandasDataset object (sktime#6668)

commit a0d950d
Merge: 2d77fd6 1fa8cfc
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Wed Jul 17 22:03:27 2024 +0530

    Merge branch 'main' into step3-4_categorical_support

commit 2d77fd6
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sat Jul 13 18:25:44 2024 +0530

    test error fix

commit 194ec28
Merge: 143a176 23bbb9b
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sat Jul 13 17:16:00 2024 +0530

    Merge updated branch 'feature_kind' with timedelta dtype added

commit 143a176
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sat Jul 13 17:11:59 2024 +0530

    added tests for categorical

commit 23bbb9b
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Fri Jul 12 12:31:52 2024 +0530

    added timedelta to DATETIME category

commit f926c58
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Fri Jul 12 10:40:29 2024 +0530

    added ignores-exog-X check to test for invalid X input

commit 460caba
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Fri Jul 12 09:32:09 2024 +0530

    added checks in transformers

commit 08aefa2
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Thu Jul 11 13:16:09 2024 +0530

    added categorical checks in other base classes

commit 0812220
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Tue Jul 9 13:29:22 2024 +0530

    linting blank line fix

commit 5f3be53
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Tue Jul 9 13:16:07 2024 +0530

    added feature_kind to req metadata in early classifiers

commit 1d7e443
Merge: 7377837 b1f6167
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Tue Jul 9 13:11:34 2024 +0530

    Merge branch updated 'feature_kind' with xarray and dask metadata support

commit b1f6167
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Tue Jul 9 13:00:36 2024 +0530

    added metadata for xarray and dask with tests

commit 0905bbb
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Mon Jul 8 13:23:36 2024 +0530

    dataloader test error workaround

commit 7377837
Merge: d36e05a 8652ba9
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Mon Jul 8 12:33:23 2024 +0530

    Merge branch updated 'feature_kind' with new bool mapping

commit 8652ba9
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Mon Jul 8 12:32:22 2024 +0530

    changed mapping of boolean dtypekind to float

commit d36e05a
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sun Jul 7 16:56:21 2024 +0530

    moved handle_categorical call to after check_is_error

commit 403ee6e
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sun Jul 7 16:52:39 2024 +0530

    indexing fix

commit a006004
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sun Jul 7 16:52:39 2024 +0530

    indexing fix

commit 3696f7b
Author: Abhay-Lejith <Abhay-Lejith@users.noreply.github.com>
Date:   Sun Jul 7 10:17:26 2024 +0000

    [AUTOMATED] update CONTRIBUTORS.md

commit 05c452c
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sun Jul 7 15:46:37 2024 +0530

    updated tag name

commit 4b557c6
Merge: 8cd48ed 978263e
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sun Jul 7 15:43:14 2024 +0530

    Merge updated branch 'categorical_tag' with class for new tag

commit 978263e
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sun Jul 7 15:41:19 2024 +0530

    added class for tag

commit 8cd48ed
Merge: 90c9f31 c1af183
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sun Jul 7 13:19:34 2024 +0530

    Merge updated branch 'feature_kind' with table support

commit c1af183
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sun Jul 7 13:07:56 2024 +0530

    extended to table scitype along with tests

commit 90c9f31
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sun Jul 7 11:19:35 2024 +0530

    calling _handle_categorical in forecasting, classification, regression

commit 655f6e0
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sun Jul 7 11:19:08 2024 +0530

    added categorical handling logic

commit db94559
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Thu Jul 4 10:29:48 2024 +0530

    added docstring and modified code acc to suggestion

commit 1321e1e
Merge: de6c92b 0eb9231
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Tue Jul 2 13:11:12 2024 +0530

    Merge pr sktime#5886 into step3-4_categorical_support

commit de6c92b
Merge: e376768 3f62d98
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Tue Jul 2 13:09:50 2024 +0530

    Merge branch 'feature_kind' into step3-4_categorical_support

commit e376768
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Tue Jul 2 11:06:07 2024 +0530

    added categorical tag

commit 3f62d98
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Mon Jul 1 11:08:47 2024 +0530

    added separate tests for testing with categorical data

commit e806cc1
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Mon Jun 24 11:09:08 2024 +0530

    renamed fields and rmeoved use of pandas func for numpy based mtypes

commit 7a58087
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Wed Jun 12 18:33:18 2024 +0530

    added simple_feature_kind and reverted changes to examples

commit ee0bc1c
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Mon Jun 3 12:02:30 2024 +0530

    changed dtype to float in panel examples which I missed earlier

commit 2f84b24
Merge: 1a64d2f e22d8dc
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Mon Jun 3 10:11:30 2024 +0530

    Merge branch 'main' into feature_kind

commit 1a64d2f
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Mon Jun 3 10:07:19 2024 +0530

    changed dtype to float in panel test examples

commit dd81851
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Sat Jun 1 16:20:17 2024 +0530

    added dtypekind metadata for panel and hierarchical

commit 83b6fe8
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Fri May 31 10:25:28 2024 +0530

    using pandas api functions to detect dtype

commit c0962f9
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Thu May 30 12:17:51 2024 +0530

    added function for getting dtypekind of series

commit 1e2110a
Merge: 898b839 6f42855
Author: Franz Király <f.kiraly@ucl.ac.uk>
Date:   Wed May 29 13:54:43 2024 +0100

    Merge branch 'main' into pr/6490

commit 898b839
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Wed May 29 11:04:28 2024 +0530

    using dtypekind enum values

commit ee8e0c4
Author: Abhay-Lejith <abhaylejith@gmail.com>
Date:   Tue May 28 14:50:43 2024 +0530

    added feature_kind metadata in series check

commit 0eb9231
Author: Anirban Ray <39331844+yarnabrina@users.noreply.github.com>
Date:   Fri Feb 16 23:37:36 2024 +0530

    @fkiraly suggestion: add rule to differentiate nested_univ and Panel

commit 0e28842
Author: Anirban Ray <39331844+yarnabrina@users.noreply.github.com>
Date:   Fri Feb 16 10:46:16 2024 +0530

    support "object" dtype in Panel

    (cherry picked from commit 1477012)

commit 25eb6f5
Author: Anirban Ray <39331844+yarnabrina@users.noreply.github.com>
Date:   Fri Feb 16 10:42:17 2024 +0530

    @fkiraly edit: add rule to differentiate nested_univ and pd.DataFrame

    Ref. sktime#5886 (review)

    (cherry picked from commit 2cecd6b)

commit a729843
Author: Anirban Ray <39331844+yarnabrina@users.noreply.github.com>
Date:   Sat Feb 3 18:29:15 2024 +0530

    added test

    (cherry picked from commit 2de7d8c)

commit c624e42
Author: Anirban Ray <39331844+yarnabrina@users.noreply.github.com>
Date:   Sat Feb 3 18:28:47 2024 +0530

    support "object" dtype in Series

    (cherry picked from commit cc3d056)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fkiraly fkiraly fkiraly approved these changes

@achieveordie achieveordie Awaiting requested review from achieveordie

@benHeid benHeid Awaiting requested review from benHeid benHeid is a code owner

@yarnabrina yarnabrina Awaiting requested review from yarnabrina yarnabrina is a code owner

Assignees

No one assigned

Labels

enhancement Adding new functionality module:datatypes datatypes module: data containers, checkers & converters

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[ENH] Adding PandasDataset from GluonTS support in Sktime

2 participants

@shlok191 @fkiraly