Storage from_file method by vlasenkov · Pull Request #1821 · pytorch/pytorch

vlasenkov · 2017-06-16T12:38:30Z

Resolves #884

torch/csrc/generic/StorageMethods.cpp

+    return NULL;
+  }
+#if defined(__APPLE__)
+  shared_mem = 0;


torch/csrc/generic/StorageMethods.cpp

+  const char *filename;
+  Py_ssize_t size = 0;
+  int shared = 0, shared_mem = 0;
+  static char *kwlist[] = {"filename", "shared", "size", "shared_mem", NULL};


torch/csrc/generic/StorageMethods.cpp

+  Py_ssize_t size = 0;
+  int shared = 0, shared_mem = 0;
+  static char *kwlist[] = {"filename", "shared", "size", "shared_mem", NULL};
+  if (!PyArg_ParseTupleAndKeywords(args, keywds, "s|pnp", kwlist,


test/test_torch.py

+        def check(filename, share_mem):
+            s1 = torch.FloatStorage.from_file(filename, True, size, share_mem)
+            t1 = torch.FloatTensor(s1).copy_(torch.randn(size))
+            t1_copy = t1.new().resize_as_(t1).copy_(t1)


test/test_torch.py

+            s2 = torch.FloatStorage.from_file(filename, True, size)
+            t2 = torch.FloatTensor(s2)
+            t2_copy = t2.new().resize_as_(t2).copy_(t2)
+            self.assertEqual(t1_copy, t2_copy, 0)


test/test_torch.py

+            self.assertEqual(t1, t2, 0)
+
+            if sys.startswith('win'):
+                os.remove(filename)


test/test_torch.py

+    def test_from_file(self):
+        size = 10000
+        filename = 'testPytorchStorageFromFile'
+        sys = platform.system()


test/test_torch.py

+
+        # check shm_open + open
+        if sys != 'Darwin':  # OS X has no /dev/shm
+            check(filename, True)


test/test_torch.py

+            t1 = torch.FloatTensor(s1).copy_(torch.randn(size))
+            t1_copy = t1.new().resize_as_(t1).copy_(t1)
+            if share_mem and sys == 'Linux':
+                filename = os.path.join('/dev/shm', filename)


apaszke · 2017-06-16T13:43:29Z

Actually now I think it should use libshm even now. Otherwise the unlinking semantics are going to change, and later commits will be breaking.

vlasenkov · 2017-06-16T13:45:48Z

So, I'm removing shared_mem for now to save at least ordinary mmap functionality

apaszke · 2017-06-16T13:51:15Z

Sounds good

vlasenkov · 2017-06-16T13:54:20Z

where should be Storage docs stored? In the _torch_docs.py, or some new _storage_docs.py?

apaszke · 2017-06-16T13:56:21Z

I think you need to create _storage_docs.py. We only document functions implemented in Python at the moment

vlasenkov · 2017-06-16T16:34:41Z

@apaszke, I've added docs. ready for review/merge

apaszke · 2017-06-16T22:34:37Z

Thank you!

…2b732d (pytorch#17264) Summary: Pull Request resolved: pytorch#17264 Previous import was 822d8df0a2a32233c6022f50a158817a0f19bdc7 Included changes: - **[4c091e0](onnx/onnx@4c091e0)**: Support defined ONNX_ML in parent cmake files (pytorch#1821) <Lu Fang> - **[57372f3](onnx/onnx@57372f3)**: Delete OpsetVersionConverter.md which is a duplicate of VersionConverter.md (pytorch#1818) <Prasanth Pulavarthi> - **[ab1c57e](onnx/onnx@ab1c57e)**: [ONNXIFI]Add extension to be implementable (pytorch#1796) <Rui Zhu> - **[b92eee8](onnx/onnx@b92eee8)**: Revert "Implement Op Annotation's for ONNX (pytorch#1648)" (pytorch#1812) <Ke Zhang> - **[61f1e9e](onnx/onnx@61f1e9e)**: Enable ONNX_ML by default (pytorch#1810) <Shinichiro Hamaji> - **[4f064a1](onnx/onnx@4f064a1)**: fix Greater and Less doc (pytorch#1811) <Guoliang Hua> - **[0628582](onnx/onnx@0628582)**: Implement Op Annotation's for ONNX (pytorch#1648) <Armen> - **[ad9d2f7](onnx/onnx@ad9d2f7)**: Versioning doc update for Opset 9 (pytorch#1805) <Vinitra Swamy> - **[e71e3be](onnx/onnx@e71e3be)**: add dilation case for ConvTranspose op (pytorch#1797) <Randy> Differential Revision: D14135024 fbshipit-source-id: 9ee7d39a5efea5d9b3c12ac3e6acc32ae83c1d0e

…2b732d (pytorch#17264) Summary: Pull Request resolved: pytorch#17264 Previous import was 822d8df0a2a32233c6022f50a158817a0f19bdc7 Included changes: - **[4c091e0](onnx/onnx@4c091e0)**: Support defined ONNX_ML in parent cmake files (pytorch#1821) <Lu Fang> - **[57372f3](onnx/onnx@57372f3)**: Delete OpsetVersionConverter.md which is a duplicate of VersionConverter.md (pytorch#1818) <Prasanth Pulavarthi> - **[ab1c57e](onnx/onnx@ab1c57e)**: [ONNXIFI]Add extension to be implementable (pytorch#1796) <Rui Zhu> - **[b92eee8](onnx/onnx@b92eee8)**: Revert "Implement Op Annotation's for ONNX (pytorch#1648)" (pytorch#1812) <Ke Zhang> - **[61f1e9e](onnx/onnx@61f1e9e)**: Enable ONNX_ML by default (pytorch#1810) <Shinichiro Hamaji> - **[4f064a1](onnx/onnx@4f064a1)**: fix Greater and Less doc (pytorch#1811) <Guoliang Hua> - **[0628582](onnx/onnx@0628582)**: Implement Op Annotation's for ONNX (pytorch#1648) <Armen> - **[ad9d2f7](onnx/onnx@ad9d2f7)**: Versioning doc update for Opset 9 (pytorch#1805) <Vinitra Swamy> - **[e71e3be](onnx/onnx@e71e3be)**: add dilation case for ConvTranspose op (pytorch#1797) <Randy> Reviewed By: yinghai Differential Revision: D14135024 fbshipit-source-id: 1e4f9dda89abf48994798d080dd5d58207a6e4b6

…pytorch#1821) remove `xfail` from `batch_norm_with_update` op in `test_grad` and `test_vmap_autograd_grad` these tests are passed now Fixes https://ontrack-internal.amd.com/browse/SWDEV-472564 cherry-picked from rocm6.3_internal_testing PR ROCm#1776

…pytorch#1821) remove `xfail` from `batch_norm_with_update` op in `test_grad` and `test_vmap_autograd_grad` these tests are passed now Fixes https://ontrack-internal.amd.com/browse/SWDEV-472564 cherry-picked from rocm6.3_internal_testing PR ROCm#1776 (cherry picked from commit 99b0758)

vlasenkov added 2 commits June 15, 2017 20:48

storage.from_file

35173d3

tests

8a1e4b0

apaszke reviewed Jun 16, 2017

View reviewed changes

soumith added the in progress label Jun 16, 2017

rm shm

1675898

vlasenkov added 4 commits June 16, 2017 17:42

storage docs

68c7b6a

storage docs

6a11203

storage docs

8dea581

fix flake8 errors

8904d95

apaszke approved these changes Jun 16, 2017

View reviewed changes

apaszke merged commit 3cecdf8 into pytorch:master Jun 16, 2017

soumith removed the in progress label Jun 16, 2017

vlasenkov deleted the storage-from-file branch July 16, 2017 13:30

houseroad mentioned this pull request Feb 19, 2019

Automatic update of fbcode/onnx to 4c091e048ca42682d63ccd3c1811560bc12b732d #17264

Closed

ezyang added the open source label Jun 24, 2019

Conversation

vlasenkov commented Jun 16, 2017

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

apaszke commented Jun 16, 2017

Uh oh!

vlasenkov commented Jun 16, 2017

Uh oh!

apaszke commented Jun 16, 2017

Uh oh!

vlasenkov commented Jun 16, 2017

Uh oh!

apaszke commented Jun 16, 2017

Uh oh!

vlasenkov commented Jun 16, 2017

Uh oh!

apaszke commented Jun 16, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants