Skip to content

Fix flaky atomic iter test#7649

Merged
bddppq merged 1 commit intopytorch:masterfrom
bddppq:relax-stats-tests
May 18, 2018
Merged

Fix flaky atomic iter test#7649
bddppq merged 1 commit intopytorch:masterfrom
bddppq:relax-stats-tests

Conversation

@bddppq
Copy link
Contributor

@bddppq bddppq commented May 17, 2018

The atomic_iter key is not necessary at the first position.
Failure reproducible in #7566 with docker image 308535385114.dkr.ecr.us-east-1.amazonaws.com/caffe2/py2-gcc5-ubuntu16.04:116-build-test-4542 via minimal test sets
pytest -s -v /code/caffe2/python/checkpoint_test.py /code/caffe2/python/hypothesis_test.py::TestOperators::test_atomic_iter_with_concurrent_steps. An inspection into the stats_key shows:

['temp_node/time_ns/count' 'temp_node/time_ns/sum'
'atomic_iter/stats/iter/num_iter'
'trainer_2/pipe/inc_total/builtin_function_or_method/NoOutput/time_ns/count'
'trainer_2/time_ns/count'
'trainer_2/pipe/inc_total/builtin_function_or_method/NoOutput/time_ns/sum'
'trainer_2/time_ns/sum'
'trainer_1/pipe/inc_total/builtin_function_or_method/NoOutput/time_ns/sum'
'trainer_0/time_ns/count' 'trainer_1/time_ns/count'
'trainer_1/pipe/inc_total/builtin_function_or_method/NoOutput/time_ns/count'
'trainer_0/time_ns/sum'
'trainer_0/pipe/inc_total/builtin_function_or_method/NoOutput/time_ns/sum'
'trainer_0/pipe/inc_total/builtin_function_or_method/NoOutput/time_ns/count'
'trainer_1/time_ns/sum']

@bddppq bddppq force-pushed the relax-stats-tests branch from 52f8584 to ef34995 Compare May 17, 2018 18:40
@bddppq
Copy link
Contributor Author

bddppq commented May 17, 2018

pr/caffe2-py2-cuda9.0-cudnn7-aten failures are compilation errors, so can not be related.

Copy link
Contributor

@xianjiec xianjiec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@yinghai yinghai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bddppq bddppq merged commit bc4feab into pytorch:master May 18, 2018
@bddppq bddppq deleted the relax-stats-tests branch May 18, 2018 04:17
petrex pushed a commit to petrex/pytorch that referenced this pull request May 23, 2018
…e2_core_hip

* 'caffe2_core_hip' of github.com:petrex/pytorch:
  Refactor cmake files to not add hip includes and libs globally
  Fix UB when converting negative floating values to uint8_t (pytorch#7644)
  [auto] Update onnx to 3fc5f43 - move finalize function to be public. (onnx/onnx#987) onnx/onnx@3fc5f43
  Fix duplicated thread pool flag
  [caffe2] Fix warning in net_async_tracing.cc (pytorch#7646)
  [auto] Update onnx to 8d548e2 - Update shape inference methods to throw exception (onnx/onnx#986) onnx/onnx@8d548e2
  Fix flaky atomic iter test (pytorch#7649)
  Fix onnx Pow export (pytorch#7657)
  [auto] Update onnx to 8356ad5 - Add unit test framework for the project C++ APIs (onnx/onnx#763) onnx/onnx@8356ad5
  [caffe2][nomnigraph] Add registry for optimization passes (pytorch#7656)
  [auto] Update onnx to 94ca052 - Update mypy version (onnx/onnx#968) onnx/onnx@94ca052
  fix nccl distributed documentation
weiyangfb pushed a commit to weiyangfb/pytorch that referenced this pull request Jun 11, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants