Skip to content

Fix DTD prototype dataset on Windows#5227

Merged
pmeier merged 10 commits intopytorch:mainfrom
pmeier:datasets/windows-ci
Jan 31, 2022
Merged

Fix DTD prototype dataset on Windows#5227
pmeier merged 10 commits intopytorch:mainfrom
pmeier:datasets/windows-ci

Conversation

@pmeier
Copy link
Contributor

@pmeier pmeier commented Jan 20, 2022 •
edited
Loading

This is the second time we are alerted by the torchdata folks that something broke, but without our CI is picking up on it. Both times it was related to Windows and we didn't notice, because we only run the prototype tests on a Linux machine.

I'm wondering if we should switch to a Windows machine for prototype tests in CI given that AFAIK no one of us is developing on Windows. Thoughts @datumbox @NicolasHug ?

cc @pmeier @bjuncek

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Jan 20, 2022 •
edited
Loading

💊 CI failures summary and remediations

As of commit 3159f66 (more details on the Dr. CI page):

  • 2/2 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See GitHub Actions build CodeQL / build (1/1)

Step: "Build TorchVision" (full log | diagnosis details | 🔁 rerun)

2022-01-31T17:01:16.1616459Z ##[error]Process completed with exit code 1. 
2022-01-31T17:01:16.0010808Z     self.finalize_options()
2022-01-31T17:01:16.0011288Z   File "/home/runner/.local/lib/python3.8/site-packages/setuptools/command/develop.py", line 52, in finalize_options
2022-01-31T17:01:16.0011776Z     easy_install.finalize_options(self)
2022-01-31T17:01:16.0012644Z   File "/home/runner/.local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 276, in finalize_options
2022-01-31T17:01:16.0013022Z     self._fix_install_dir_for_user_site()
2022-01-31T17:01:16.0013560Z   File "/home/runner/.local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 382, in _fix_install_dir_for_user_site
2022-01-31T17:01:16.0013933Z     self.create_home_path()
2022-01-31T17:01:16.0014676Z   File "/home/runner/.local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 1338, in create_home_path
2022-01-31T17:01:16.0015069Z     if path.startswith(home) and not os.path.isdir(path):
2022-01-31T17:01:16.0015489Z AttributeError: 'int' object has no attribute 'startswith'
2022-01-31T17:01:16.1616459Z ##[error]Process completed with exit code 1.
2022-01-31T17:01:16.1673198Z Post job cleanup.
2022-01-31T17:01:16.3166077Z [command]/usr/bin/git version
2022-01-31T17:01:16.3228584Z git version 2.34.1
2022-01-31T17:01:16.3279658Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-01-31T17:01:16.3338330Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-01-31T17:01:16.3762140Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-01-31T17:01:16.3814274Z http.https://github.com/.extraheader
2022-01-31T17:01:16.3829279Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-01-31T17:01:16.3892289Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-01-31T17:01:16.4547416Z Cleaning up orphan processes

1 failure not recognized by patterns:

Job Step Action
CircleCI cmake_macos_cpu curl -o conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
sh conda.sh -b
source $HOME/miniconda3/bin/activate
conda install -yq conda-build cmake
packaging/build_cmake.sh
🔁 rerun
This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@pmeier pmeier changed the title [DEBUG] run prototype tests on Windows Fix line endings in prototype datasets mock data generation Jan 20, 2022
@pmeier pmeier requested a review from NicolasHug January 20, 2022 08:52
@pmeier pmeier marked this pull request as ready for review January 20, 2022 08:52
@pmeier
Copy link
Contributor Author

pmeier commented Jan 20, 2022

Turns out, this is actually an issue in torchdata with properly stripping the newline characters: meta-pytorch/data#173.

@pmeier pmeier marked this pull request as draft January 20, 2022 09:28
@pmeier
Copy link
Contributor Author

pmeier commented Jan 20, 2022

With my fix in meta-pytorch/data#174, we do not need to care about the line endings while generating or reading the files. Thus, this PR only fixes the mock data generation of DTD.

@pmeier pmeier changed the title Fix line endings in prototype datasets mock data generation Fix DTD prototype mock data generation Jan 20, 2022
@pmeier pmeier marked this pull request as ready for review January 20, 2022 09:44
@pmeier pmeier mentioned this pull request Jan 20, 2022
Closed
@pmeier pmeier changed the title Fix DTD prototype mock data generation Fix DTD prototype dataset on Windows Jan 31, 2022
NicolasHug
NicolasHug approved these changes Jan 31, 2022
View reviewed changes
Copy link
Member

@NicolasHug NicolasHug left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pmeier

@pmeier pmeier merged commit 8aad0e0 into pytorch:main Jan 31, 2022
@pmeier pmeier deleted the datasets/windows-ci branch January 31, 2022 17:18
facebook-github-bot pushed a commit that referenced this pull request Feb 3, 2022
@yiwen-song @facebook-github-bot
[fbsync] Fix DTD prototype dataset on Windows (#5227)
1af32fe 
Summary:
* [DEBUG] run prototype tests on Windows

* disable duplicate resource class

* fix newlines in mock data generation

* fix path matching for DTD

* revert CI changes

* Revert "fix newlines in mock data generation"

This reverts commit f53cf60.

* rerun tests with fix in torchdata

* revert CI changes

* add explanation

Reviewed By: kazhang

Differential Revision: D33927486

fbshipit-source-id: 6242128b812f2c797b9c8b69d3d7b5c9bf34fd0e
@pmeier pmeier mentioned this pull request Mar 4, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@NicolasHug NicolasHug NicolasHug approved these changes

Assignees

No one assigned

Labels

ciflow/default cla signed module: datasets module: tests prototype

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pmeier @facebook-github-bot @NicolasHug