Skip to content

Remove download for ImageNet#1457

Merged
fmassa merged 15 commits intopytorch:masterfrom
pmeier:remove_imagenet_download
Oct 21, 2019
Merged

Remove download for ImageNet#1457
fmassa merged 15 commits intopytorch:masterfrom
pmeier:remove_imagenet_download

Conversation

@pmeier
Copy link
Contributor

@pmeier pmeier commented Oct 14, 2019 •
edited
Loading

addresses #1453

I introduced the function parse_train_archive and parse_val_archive and changed parse_devkit to take in an str pointing to an archive. Users that downloaded the dataset externally can use them to prepare the folders.

pmeier
pmeier commented Oct 14, 2019
View reviewed changes
pmeier
pmeier commented Oct 14, 2019
View reviewed changes
pmeier
pmeier commented Oct 14, 2019
View reviewed changes
@pmeier pmeier marked this pull request as ready for review October 15, 2019 07:00
pmeier
pmeier commented Oct 15, 2019
View reviewed changes
@pmeier
Copy link
Contributor Author

pmeier commented Oct 15, 2019

Tests currently fail since the new code raises an error if the MD5 of the archive does not match. Shall I remove the download flag in the tests and simply parse the archives at first?

vision/test/test_datasets.py

Lines 111 to 116 in ed5b2dc

@mock.patch('torchvision.datasets.utils.download_url')
@unittest.skipIf(not HAS_SCIPY, "scipy unavailable")
def test_imagenet(self, mock_download):
with imagenet_root() as root:
dataset = torchvision.datasets.ImageNet(root, split='train', download=True)
self.generic_classification_dataset_test(dataset)

fmassa
fmassa requested changes Oct 15, 2019
View reviewed changes
Copy link
Member

@fmassa fmassa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

I have a comment, I think it will make usability better. Let me know what you think

pmeier
pmeier commented Oct 16, 2019
View reviewed changes
@fmassa
Copy link
Member

fmassa commented Oct 16, 2019

@pmeier thanks for the PR! Can you rebase your changes on top of master, I've fixed CI for most of them.

@pmeier pmeier force-pushed the remove_imagenet_download branch from 61d7e23 to 2548910 Compare October 16, 2019 13:34
@pmeier pmeier requested a review from fmassa October 17, 2019 14:56
@fmassa
Copy link
Member

fmassa commented Oct 18, 2019

LGTM, but test failures seem to be related?

_____________________________ Tester.test_imagenet _____________________________

self = <test_datasets.Tester testMethod=test_imagenet>
mock_check = <MagicMock name='_verify_archive' id='140159361171976'>

    @mock.patch('torchvision.datasets.imagenet.ImageNet._verify_archive')
    @unittest.skipIf(not HAS_SCIPY, "scipy unavailable")
    def test_imagenet(self, mock_check):
        with imagenet_root() as root:
>           dataset = torchvision.datasets.ImageNet(root, split='train')

test/test_datasets.py:115: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.5/site-packages/torchvision/datasets/imagenet.py:62: in __init__
    self.extract_archives()
../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.5/site-packages/torchvision/datasets/imagenet.py:81: in extract_archives
    parse_devkit_archive(archive)
../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.5/site-packages/torchvision/datasets/imagenet.py:154: in parse_devkit_archive
    _verify_archive(archive, ARCHIVE_DICT["devkit"]["md5"], force)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

archive = '/tmp/tmpzh1hy800/ILSVRC2012_devkit_t12.tar.gz'
md5 = 'fa75699e90414af021442c21a62c3abf', force = False

    def _verify_archive(archive, md5, force):
        if not check_integrity(archive):
            raise RuntimeError("The file {} doesn't exist.".format(archive))
        if not check_md5(archive, md5) and not force:
            msg = ("The MD5 checksum of the file {} and the original archive do not match. "
                   "Use force=True to force an extraction")
>           raise RuntimeError(msg.format(archive))
E           RuntimeError: The MD5 checksum of the file /tmp/tmpzh1hy800/ILSVRC2012_devkit_t12.tar.gz and the original archive do not match. Use force=True to force an extraction

pmeier
pmeier commented Oct 18, 2019
View reviewed changes
* unify _verify_archive() method and function
eb1bd7a 
* remove force flag for parse_*_archive functions
* cleanup
@codecov-io
Copy link

codecov-io commented Oct 21, 2019 •
edited
Loading

Codecov Report

Merging #1457 into master will decrease coverage by 0.11%.
The diff coverage is 80.89%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1457      +/-   ##
==========================================
- Coverage   64.46%   64.34%   -0.12%     
==========================================
  Files          83       83              
  Lines        6421     6454      +33     
  Branches      982      992      +10     
==========================================
+ Hits         4139     4153      +14     
- Misses       1992     2006      +14     
- Partials      290      295       +5
Impacted Files Coverage Δ
torchvision/datasets/imagenet.py 82.14% <80.89%> (-8.77%) ⬇️
torchvision/datasets/utils.py 58.38% <0%> (-3.73%) ⬇️
torchvision/transforms/functional_tensor.py 57.14% <0%> (-2.86%) ⬇️
torchvision/io/video.py 75% <0%> (-0.33%) ⬇️
torchvision/models/detection/transform.py 66.92% <0%> (ø) ⬆️
torchvision/models/detection/roi_heads.py 56.14% <0%> (+0.37%) ⬆️
torchvision/ops/boxes.py 100% <0%> (+5.55%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e279eba...b57b90e. Read the comment docs.

fmassa
fmassa approved these changes Oct 21, 2019
View reviewed changes
Copy link
Member

@fmassa fmassa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@fmassa fmassa merged commit f46f2c1 into pytorch:master Oct 21, 2019
@pmeier pmeier deleted the remove_imagenet_download branch October 21, 2019 11:49
@pmeier pmeier mentioned this pull request Oct 21, 2019
Closed
@z-a-f z-a-f mentioned this pull request Nov 8, 2019
Closed
@kanonjz
Copy link

kanonjz commented Dec 7, 2019

I downloaded imagenet myself and used parse_val_archive to prepare the folders, but got an error below. What is the meta.bin? I didn't find it in the imagenet.

The meta file meta.bin is not present in the root directory or is corrupted. " "This file is automatically created by the ImageNet dataset.

@pmeier pmeier mentioned this pull request Dec 7, 2019
Closed
@pmeier
Copy link
Contributor Author

pmeier commented Dec 7, 2019

@kanonjz I think your question justifies an independent issue. I opened one for you. Have a look at #1646. I answered your question there.

@fmassa fmassa mentioned this pull request Apr 15, 2020
Merged
@NicolasHug NicolasHug mentioned this pull request Mar 11, 2021
Merged
@pmeier pmeier mentioned this pull request May 20, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fmassa fmassa fmassa approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@pmeier @fmassa @codecov-io @kanonjz